the UIA functionality cannot retrieve the control description.

[Davider]

I am using an open-source Python component that locates controls through the control descriptions in UIA. The related code link is as follows:
https://github.com/GuanYixuan/pyJianYing...ler.py#L92

I tried to perform the search in the "Find UI Elements" dialog of LA, but I wasn't successful (it may require some options for control). Below is a demonstration of the operation.
https://download.ru/files/hfGQ59NB
 The software used in the demonstration: https://www.capcut.com/

Thanks in advance for any suggestions and help

[Gintaras]

Will be improved in next LA. Thank you.

[Davider]

How to use general C# code to implement the functionality of the Python code? The following code was unsuccessful.
 

Code:

Copy      Help

// NuGet Package References: Interop.UIAutomationClient
using System;
using System.Runtime.InteropServices;
using Interop.UIAutomationClient;

void Main()
{
    var controller = new JianyingController();
    try
    {
        controller.OpenDraft("3月8日");
    }
    catch (Exception ex)
    {
        Console.WriteLine($"Error: {ex.Message}");
    }
    finally
    {
        controller.Dispose();
    }
}

public class ControlFinder
{
    public static IUIAutomationCondition CreateDescriptionMatcher(
        IUIAutomation automation,
        string targetDesc,
        int depth = 2,
        bool exact = false)
    {
        targetDesc = targetDesc.ToLower();
        
        // Use UIA_FullDescriptionPropertyId (30159)
        var propertyId = 30159;
        var propertyCondition = automation.CreatePropertyCondition(
            propertyId,
            targetDesc
        );
        
        return propertyCondition;
    }
}

public class JianyingController : IDisposable
{
    private readonly IUIAutomation _automation;
    private IUIAutomationElement _appWindow;
    
    public JianyingController()
    {
        // Create an instance of CUIAutomation
        _automation = new CUIAutomation();
        _appWindow = FindJianyingWindow();
        
        if (_appWindow == null)
        {
            throw new Exception("Failed to find the Jianying window.");
        }
    }
    
    private IUIAutomationElement FindJianyingWindow()
    {
        // Get the desktop root element
        var rootElement = _automation.GetRootElement();
        
        // Create search condition: find by window name
        var condition = _automation.CreatePropertyCondition(
            UIA_PropertyIds.UIA_NamePropertyId,
            "剪映专业版"
        );
        
        // Find the Jianying window
        return rootElement.FindFirst(Interop.UIAutomationClient.TreeScope.TreeScope_Children, condition);
    }
    
    public void OpenDraft(string draftName)
    {
        Console.WriteLine($"Attempting to open the draft: {draftName}");
        
        // Create a search condition for the draft title
        var draftTitleCondition = _automation.CreatePropertyCondition(
            UIA_PropertyIds.UIA_AutomationIdPropertyId,
            $"HomePageDraftTitle:{draftName}"
        );
        
        // Search for the draft element
        var draftElement = _appWindow.FindFirst(
            Interop.UIAutomationClient.TreeScope.TreeScope_Descendants,
            draftTitleCondition
        );
        
        if (draftElement == null)
        {
            throw new Exception($"Failed to find the draft named {draftName}.");
        }

        // Get the parent button element
        var walker = _automation.ControlViewWalker;
        var draftButton = walker.GetParentElement(draftElement);

        if (draftButton == null)
        {
            throw new Exception("Failed to find the parent button element of the draft.");
        }

        // Get the Invoke Pattern and click
        var invokePattern = draftButton.GetCurrentPattern(UIA_PatternIds.UIA_InvokePatternId)
            as IUIAutomationInvokePattern;

        if (invokePattern == null)
        {
            throw new Exception("Failed to retrieve the Invoke pattern of the button.");
        }

        invokePattern.Invoke();
    }

    public void Dispose()
    {
        if (_appWindow != null)
        {
            Marshal.ReleaseComObject(_appWindow);
            _appWindow = null;
        }

        if (_automation != null)
        {
            Marshal.ReleaseComObject(_automation);
        }
    }
}

// Common UIA Property IDs
public static class UIA_PropertyIds
{
    public const int UIA_NamePropertyId = 30005;
    public const int UIA_AutomationIdPropertyId = 30011;
    public const int UIA_FullDescriptionPropertyId = 30159;
}

// Common UIA Pattern IDs
public static class UIA_PatternIds
{
    public const int UIA_InvokePatternId = 10000;
}

python code:

Code:

Copy      Help

import uiautomation as uia

class ControlFinder:
    """Control Finder that encapsulates logic related to control search"""

    @staticmethod
    def desc_matcher(target_desc: str, depth: int = 2, exact: bool = False) -> Callable[[uia.Control, int], bool]:
        """Matcher function to find controls based on full_description"""
        target_desc = target_desc.lower()
        def matcher(control: uia.Control, _depth: int) -> bool:
            if _depth != depth:
                return False
            full_desc: str = control.GetPropertyValue(30159).lower()
            return (target_desc == full_desc) if exact else (target_desc in full_desc)
        return matcher

class Jianying_controller:
    """Jianying Controller"""

    app: uia.WindowControl
    """Jianying Window"""

    def open_draft(self, draft_name: str) -> None:
        """Open the specified Jianying draft

        Args:
            draft_name (`str`): The name of the Jianying draft to be opened
        """
        print(f"Starting to open {draft_name}")

        # Click the corresponding draft
        draft_name_text = self.app.TextControl(
            searchDepth=2,
            Compare=ControlFinder.desc_matcher(f"HomePageDraftTitle:{draft_name}", exact=True)
        )
        if not draft_name_text.Exists(0):
            raise exceptions.DraftNotFound(f"Could not find the Jianying draft named {draft_name}")
        draft_btn = draft_name_text.GetParentControl()
        assert draft_btn is not None
        draft_btn.Click(simulateMove=False)

[Gintaras]

Using FlaUI
 

Code:

Copy      Help

// script "test FlaUI.cs"
/*/ nuget FlaUI\FlaUI.UIA3; /*/

using FlaUI.UIA3;
using FlaUI.Core.Conditions;

using var uia = new UIA3Automation();

var w1 = wnd.find(1, "CapCut", "Qt622QWindowIcon");
var ew1 = uia.FromHandle((nint)w1);

var condition = new PropertyCondition(uia.PropertyLibrary.Element.FullDescription, "HomePageDraftTitle:0319");
var e2 = ew1.FindFirstDescendant(condition) ?? throw new NotFoundException();
print.it(e2);

e2.Click();

[Davider]

Thanks for your help, the code works very well.
In the next LA version, will it only take two lines of code to implement the above functionality?
It would be more convenient and elegant without using the FlaUI component. The FlaUI component requires referencing five DLLs.

[Davider]

If possible, supporting XPath for locating elements would be more versatile, similar to the following test code for the FlaUI component.
https://github.com/FlaUI/FlaUI/blob/mast...thTests.cs

[Gintaras]

In the next LA version, will it only take two lines of code to implement the above functionality?

The same code as now.

XPath unlikely. The path syntax

Code:

Copy      Help

[role1, name1, etc][role2, name2, etc]

will not change.

[Gintaras]

Using just Interop.UIAutomationClient from FlaUI:

Code:

Copy      Help

// script "test Interop.UIAutomationClient.cs"
/*/ nuget -\Interop.UIAutomationClient; /*/

using UIA = Interop.UIAutomationClient;

var uia = new UIA.CUIAutomation8Class() as UIA.IUIAutomation;

var w1 = wnd.find(1, "CapCut", "Qt622QWindowIcon");
var ew1 = uia.ElementFromHandle((nint)w1);

var condition = uia.CreatePropertyConditionEx(30159, "HomePageDraftTitle:0319", 0);
var e2 = ew1.FindFirst(UIA.TreeScope.TreeScope_Descendants, condition) ?? throw new NotFoundException();

e2.GetClickablePoint(out var p2);
mouse.click(p2.x, p2.y);

[Davider]

Thank you for sharing.
#9 code only requires a single DLL.
 

The same code as now.

 Does this mean that there are no plans to add UIA3 support to LA, and an external component must be used to achieve the above functionality?

---

#9 code:I need to click on the parent object of the e2 control. The following code will throw an error.

err info:
System.NullReferenceException: Object reference not set to an instance of an object.
 

Code:

Copy      Help

var e3 = e2.GetCachedParent();
e3.GetClickablePoint(out var p2);
mouse.click(p2.x, p2.y);

[Gintaras]

LA always used UIA3, and nothing will change, only the "get description" code will be fixed.

[Davider]

Thanks for your clarification again.
As I understand it, if the "Find UI Element" dialog detects the Description property, then it should be possible to achieve the above functionality without using the FlaUI component.

[Gintaras]

Yes. The dialog displays all non-empty properties that can be used with "find element" functions. In current version fails to get description of some elements; will work in next LA.

---

System.NullReferenceException

Use the "Current" functions (like GetCurrentParent), not the "Cached" functions (like GetCachedParent).

[Davider]

Use the "Current" functions (like GetCurrentParent)

 There is no such function. 

https://i.ibb.co/9HCX0x8b/par.jpg

[Gintaras]

Code:

Copy      Help

var treeWalker = uia.CreateTreeWalker(uia.RawViewCondition);
e2 = treeWalker.GetParentElement(e2);

[Davider]

LA is so powerful that it only takes two lines of code to get the job done.
However, there is an issue: I have to manually select UIA every time, as shown in the image below.
Suggestion: After pressing Shift + F3 once, UIA should be automatically selected.
https://i.ibb.co/nsYkXntD/aa.jpg

Code:

Copy      Help

var w1 = wnd.find(1, "", "Qt622QWindowIcon");
var e1 = w1.Elm["STATICTEXT", prop: "desc=HomePageDraftTitle:3月8日", flags: EFFlags.UIA].Find(1);
e1.Parent.MouseClick();

---

If QM can also achieve this functionality, that would be great!
Which source file contains the implementation code? I’d like to try using ChatGPT to work on it.

[Gintaras]

QM UIA examples