Skip to content

Commit a0e5085

Browse files
carlos-zamoracarlos-zamora
authored
Expose Text Attributes to UI Automation (#10336)
## Summary of the Pull Request This implements `GetAttributeValue` and `FindAttribute` for `UiaTextRangeBase` (the shared `ITextRangeProvider` for Conhost and Windows Terminal). This also updates `UiaTracing` to collect more useful information on these function calls. ## References #7000 - Epic [Text Attribute Identifiers](https://docs.microsoft.com/en-us/windows/win32/winauto/uiauto-textattribute-ids) [ITextRangeProvider::GetAttributeValue](https://docs.microsoft.com/en-us/windows/win32/api/uiautomationcore/nf-uiautomationcore-itextrangeprovider-getattributevalue) [ITextRangeProvider::FindAttribute](https://docs.microsoft.com/en-us/windows/win32/api/uiautomationcore/nf-uiautomationcore-itextrangeprovider-findattribute) ## PR Checklist * [X] Closes #2161 * [X] Tests added/passed ## Detailed Description of the Pull Request / Additional comments - `TextBuffer`: - Exposes a new `TextBufferCellIterator` that takes in an end position. This simplifies the logic drastically as we can now use this iterator to navigate through the text buffer. The iterator can also expose the position in the buffer. - `UiaTextRangeBase`: - Shared logic & helper functions: - Most of the text attributes are stored as `TextAttribute`s in the text buffer. To extract them, we generate an attribute verification function via `_getAttrVerificationFn()`, then use that to verify if a given cell has the desired attribute. - A few attributes are special (i.e. font name, font size, and "is read only"), in that they are (1) acquired differently and (2) consistent across the entire text buffer. These are handled separate from the attribute verification function. - `GetAttributeValue`: Retrieve the attribute verification of the first cell in the range. Then, verify that the entire range has that attribute by iterating through the text range. If a cell does not have that attribute, return the "reserved mixed attribute value". - `FindAttribute`: Iterate through the text range and leverage the attribute verification function to find the first contiguous range with that attribute. Then, make the end exclusive and output a `UiaTextRangeBase`. This function must be able to perform a search backwards, so we abstract the "start" and "end" into `resultFirstAnchor` and `resultSecondAnchor`, then perform post processing to output a valid `UiaTextRangeBase`. - `UiaTracing`: - `GetAttributeValue`: Log uia text range, desired attribute, resulting attribute metadata, and the type of the result. - `FindAttribute`: Log uia text range, desired attribute and attribute metadata, if we were searching backwards, the type of the result, and the resulting text range. - `AttributeType` is a nice way to understand/record if the result was either of the reserved UIA values, a normal result, or an error. - `UiaTextRangeTests`: - `GetAttributeValue`: - verify that we know which attributes we support - test each of the known text attributes (expecting 100% code coverage for `_getAttrVerificationFn()`) - `FindAttribute`: - test each of the known _special_ text attributes - test `IsItalic`. NOTE: I'm explicitly only testing one of the standard text attributes because the logic is largely the same between all of them and they leverage `_getAttrVerificationFn()`. ## Validation Steps Performed - @codeofdusk has been testing this Conhost build - Tests added for Conhost and shared implementation - Windows Terminal changes were manually verified using accessibility insights and NVDA
1 parent d57fb84 commit a0e5085

17 files changed

Lines changed: 866 additions & 46 deletions

‎src/buffer/out/textBufferCellIterator.cpp‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -265,3 +265,8 @@ const OutputCellView* TextBufferCellIterator::operator->() const noexcept
265265
{
266266
return &_view;
267267
}
268+
269+
COORD TextBufferCellIterator::Pos() const noexcept
270+
{
271+
return _pos;
272+
}

‎src/buffer/out/textBufferCellIterator.hpp‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,8 @@ class TextBufferCellIterator
4747
const OutputCellView& operator*() const noexcept;
4848
const OutputCellView* operator->() const noexcept;
4949

50+
COORD Pos() const noexcept;
51+
5052
protected:
5153
void _SetPos(const COORD newPos);
5254
void _GenerateView();

‎src/cascadia/TerminalControl/ControlCore.cpp‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -592,6 +592,8 @@ namespace winrt::Microsoft::Terminal::Control::implementation
592592
const int newDpi = static_cast<int>(static_cast<double>(USER_DEFAULT_SCREEN_DPI) *
593593
_compositionScale);
594594

595+
_terminal->SetFontInfo(_actualFont);
596+
595597
// TODO: MSFT:20895307 If the font doesn't exist, this doesn't
596598
// actually fail. We need a way to gracefully fallback.
597599
_renderer->TriggerFontChange(newDpi, _desiredFont, _actualFont);

‎src/cascadia/TerminalControl/XamlUiaTextRange.cpp‎

Lines changed: 46 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@
55
#include "XamlUiaTextRange.h"
66
#include "../types/TermControlUiaTextRange.hpp"
77
#include <UIAutomationClient.h>
8+
#include <UIAutomationCoreApi.h>
89

910
// the same as COR_E_NOTSUPPORTED
1011
// we don't want to import the CLR headers to get it
@@ -89,12 +90,52 @@ namespace winrt::Microsoft::Terminal::Control::implementation
8990

9091
winrt::Windows::Foundation::IInspectable XamlUiaTextRange::GetAttributeValue(int32_t textAttributeId) const
9192
{
92-
// Copied functionality from Types::UiaTextRange.cpp
93-
if (textAttributeId == UIA_IsReadOnlyAttributeId)
93+
// Call the function off of the underlying UiaTextRange.
94+
VARIANT result;
95+
THROW_IF_FAILED(_uiaProvider->GetAttributeValue(textAttributeId, &result));
96+
97+
// Convert the resulting VARIANT into a format that is consumable by XAML.
98+
switch (result.vt)
99+
{
100+
case VT_BSTR:
101+
{
102+
return box_value(result.bstrVal);
103+
}
104+
case VT_I4:
105+
{
106+
// Surprisingly, `long` is _not_ a WinRT type.
107+
// So we have to use `int32_t` to make sure this is output properly.
108+
// Otherwise, you'll get "Attribute does not exist" out the other end.
109+
return box_value<int32_t>(result.lVal);
110+
}
111+
case VT_R8:
112+
{
113+
return box_value(result.dblVal);
114+
}
115+
case VT_BOOL:
94116
{
95-
return winrt::box_value(false);
117+
return box_value<bool>(result.boolVal);
96118
}
97-
else
119+
case VT_UNKNOWN:
120+
{
121+
// This one is particularly special.
122+
// We might return a special value like UiaGetReservedMixedAttributeValue
123+
// or UiaGetReservedNotSupportedValue.
124+
// Some text attributes may return a real value, however, none of those
125+
// are supported at this time.
126+
// So we need to figure out what was actually intended to be returned.
127+
128+
com_ptr<IUnknown> mixedAttributeVal;
129+
UiaGetReservedMixedAttributeValue(mixedAttributeVal.put());
130+
131+
if (result.punkVal == mixedAttributeVal.get())
132+
{
133+
return Windows::UI::Xaml::DependencyProperty::UnsetValue();
134+
}
135+
136+
[[fallthrough]];
137+
}
138+
default:
98139
{
99140
// We _need_ to return XAML_E_NOT_SUPPORTED here.
100141
// Returning nullptr is an improper implementation of it being unsupported.
@@ -103,6 +144,7 @@ namespace winrt::Microsoft::Terminal::Control::implementation
103144
// Magically, this doesn't affect other forms of navigation...
104145
winrt::throw_hresult(XAML_E_NOT_SUPPORTED);
105146
}
147+
}
106148
}
107149

108150
void XamlUiaTextRange::GetBoundingRectangles(com_array<double>& returnValue) const

‎src/cascadia/TerminalCore/Terminal.cpp‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,6 @@
66
#include "../../terminal/parser/OutputStateMachineEngine.hpp"
77
#include "TerminalDispatch.hpp"
88
#include "../../inc/unicode.hpp"
9-
#include "../../inc/DefaultSettings.h"
109
#include "../../inc/argb.h"
1110
#include "../../types/inc/utils.hpp"
1211
#include "../../types/inc/colorTable.hpp"

‎src/cascadia/TerminalCore/Terminal.hpp‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@
55

66
#include <conattrs.hpp>
77

8+
#include "../../inc/DefaultSettings.h"
89
#include "../../buffer/out/textBuffer.hpp"
910
#include "../../types/inc/sgrStack.hpp"
1011
#include "../../renderer/inc/BlinkingState.hpp"
@@ -68,6 +69,7 @@ class Microsoft::Terminal::Core::Terminal final :
6869

6970
void UpdateSettings(winrt::Microsoft::Terminal::Core::ICoreSettings settings);
7071
void UpdateAppearance(const winrt::Microsoft::Terminal::Core::ICoreAppearance& appearance);
72+
void SetFontInfo(const FontInfo& fontInfo);
7173

7274
// Write goes through the parser
7375
void Write(std::wstring_view stringView);
@@ -160,6 +162,7 @@ class Microsoft::Terminal::Core::Terminal final :
160162
COORD GetTextBufferEndPosition() const noexcept override;
161163
const TextBuffer& GetTextBuffer() noexcept override;
162164
const FontInfo& GetFontInfo() noexcept override;
165+
std::pair<COLORREF, COLORREF> GetAttributeColors(const TextAttribute& attr) const noexcept override;
163166

164167
void LockConsole() noexcept override;
165168
void UnlockConsole() noexcept override;
@@ -168,7 +171,6 @@ class Microsoft::Terminal::Core::Terminal final :
168171
#pragma region IRenderData
169172
// These methods are defined in TerminalRenderData.cpp
170173
const TextAttribute GetDefaultBrushColors() noexcept override;
171-
std::pair<COLORREF, COLORREF> GetAttributeColors(const TextAttribute& attr) const noexcept override;
172174
COORD GetCursorPosition() const noexcept override;
173175
bool IsCursorVisible() const noexcept override;
174176
bool IsCursorOn() const noexcept override;
@@ -276,6 +278,10 @@ class Microsoft::Terminal::Core::Terminal final :
276278
size_t _hyperlinkPatternId;
277279

278280
std::wstring _workingDirectory;
281+
282+
// This default fake font value is only used to check if the font is a raster font.
283+
// Otherwise, the font is changed to a real value with the renderer via TriggerFontChange.
284+
FontInfo _fontInfo{ DEFAULT_FONT_FACE, TMPF_TRUETYPE, 10, { 0, DEFAULT_FONT_SIZE }, CP_UTF8, false };
279285
#pragma region Text Selection
280286
// a selection is represented as a range between two COORDs (start and end)
281287
// the pivot is the COORD that remains selected when you extend a selection in any direction

‎src/cascadia/TerminalCore/terminalrenderdata.cpp‎

Lines changed: 6 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -27,23 +27,15 @@ const TextBuffer& Terminal::GetTextBuffer() noexcept
2727
return *_buffer;
2828
}
2929

30-
// Creating a FontInfo can technically throw (on string allocation) and this is noexcept.
31-
// That means this will std::terminate. We could come back and make there be a default constructor
32-
// backup to FontInfo that throws no exceptions and allocates a default FontInfo structure.
33-
#pragma warning(push)
34-
#pragma warning(disable : 26447)
3530
const FontInfo& Terminal::GetFontInfo() noexcept
3631
{
37-
// TODO: This font value is only used to check if the font is a raster font.
38-
// Otherwise, the font is changed with the renderer via TriggerFontChange.
39-
// The renderer never uses any of the other members from the value returned
40-
// by this method.
41-
// We could very likely replace this with just an IsRasterFont method
42-
// (which would return false)
43-
static const FontInfo _fakeFontInfo(DEFAULT_FONT_FACE, TMPF_TRUETYPE, 10, { 0, DEFAULT_FONT_SIZE }, CP_UTF8, false);
44-
return _fakeFontInfo;
32+
return _fontInfo;
33+
}
34+
35+
void Terminal::SetFontInfo(const FontInfo& fontInfo)
36+
{
37+
_fontInfo = fontInfo;
4538
}
46-
#pragma warning(pop)
4739

4840
const TextAttribute Terminal::GetDefaultBrushColors() noexcept
4941
{

‎src/host/renderData.hpp‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@ class RenderData final :
2727
COORD GetTextBufferEndPosition() const noexcept override;
2828
const TextBuffer& GetTextBuffer() noexcept override;
2929
const FontInfo& GetFontInfo() noexcept override;
30+
std::pair<COLORREF, COLORREF> GetAttributeColors(const TextAttribute& attr) const noexcept override;
3031

3132
std::vector<Microsoft::Console::Types::Viewport> GetSelectionRects() noexcept override;
3233

@@ -37,8 +38,6 @@ class RenderData final :
3738
#pragma region IRenderData
3839
const TextAttribute GetDefaultBrushColors() noexcept override;
3940

40-
std::pair<COLORREF, COLORREF> GetAttributeColors(const TextAttribute& attr) const noexcept override;
41-
4241
COORD GetCursorPosition() const noexcept override;
4342
bool IsCursorVisible() const noexcept override;
4443
bool IsCursorOn() const noexcept override;

0 commit comments

Comments
 (0)