Skip to content

Commit 0050cc7

Browse files
isaacclaude
isaac
and
claude
committed
Rich-message media in gallery/shared-media/preview pipelines via Message.effectiveMedia
Add Message/EngineMessage.effectiveMedia (= message.media when non-empty, else richText.instantPage.allMedia()) and route the media-consuming sites through it so a rich message's instant-page media participates in the same pipelines as normal message.media: shared-media grids/file-rows, search media grid, gallery open + item nodes + footer, the peer audio/voice playlist, secret-media preview, resource-by-id resolution, recent downloads, downloaded-media store, delete-time resource cleanup, cache-usage stats, the in-chat download manager, and the context-menu / share actions (Save to Camera Roll, copy image, save audio/music to files). For normal messages effectiveMedia == message.media, so each swap is behavior-preserving; rich messages render their own bubble via ChatMessageRichDataBubbleContentNode (not the text/file bubbles), so those paths are deliberately untouched, as are the forward path (the attribute travels with the forward) and the markdown-based rich-edit path. First-media scope for now. See docs/instantpage-richtext.md for the full architecture + invariants. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent c95e014 commit 0050cc7

30 files changed

Lines changed: 130 additions & 64 deletions

‎CLAUDE.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -62,7 +62,7 @@ Rare exceptions: top-level view-controller views integrating with the system's f
6262

6363
## InstantPage V2 & rich-text messages
6464

65-
Typed markdown with structure the regular message-entity set can't represent (headings, lists, tables, formulas, nested blockquotes) is sent as a **rich message** — a `RichTextMessageAttribute` carrying an `InstantPage`, drawn by `ChatMessageRichDataBubbleContentNode` via the **InstantPage V2** renderer (with AI-streaming progressive reveal, inline custom emoji, and entity cases). The detailed architecture and non-obvious invariants — streaming reveal, V2 table/text-box layout, custom-emoji & entity round-trips, task-list checkboxes, nested blockquotes, thinking blocks, and the markdown send / edit / copy / paste paths — live in [`docs/instantpage-richtext.md`](docs/instantpage-richtext.md).
65+
Typed markdown with structure the regular message-entity set can't represent (headings, lists, tables, formulas, nested blockquotes) is sent as a **rich message** — a `RichTextMessageAttribute` carrying an `InstantPage`, drawn by `ChatMessageRichDataBubbleContentNode` via the **InstantPage V2** renderer (with AI-streaming progressive reveal, inline custom emoji, and entity cases). The detailed architecture and non-obvious invariants — streaming reveal, V2 table/text-box layout, custom-emoji & entity round-trips, task-list checkboxes, nested blockquotes, thinking blocks, the markdown send / edit / copy / paste paths, and surfacing rich-message media through the shared-media/gallery/preview pipelines via `Message.effectiveMedia` — live in [`docs/instantpage-richtext.md`](docs/instantpage-richtext.md).
6666

6767
## Postbox → TelegramEngine refactor (in progress)
6868

‎docs/instantpage-richtext.md‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -426,3 +426,44 @@ A server-sent rich message can arrive **partial** when the content is long: the
426426
- **`showMoreExpanded` is part of BOTH layout caches.** It is in the `currentPageLayout` cache key **and** the `pageView` content key (`pageViewMessageKey`). This is required because the cached-expand path (full page already on the attribute) performs **no postbox write**, so `stableVersion` does not bump — without the key, the cached partial layout/content would shadow the expand.
427427
- **Tap (`activateShowMore`):** if `fullInstantPage` is already cached → set expanded + `requestMessageUpdate` immediately (no network, no shimmer); otherwise shimmer the link and fetch, expanding only once the full page lands. Guards against a second in-flight request and against re-expanding.
428428
- **Expand grows the bubble downward in screen space** (top fixed) via `info?.setInvertOffsetDirection()` on the `ListViewItemApply` in the apply closure, fired only on the `appliedShowMoreExpanded → showMoreExpanded` transition (never on first apply). Same mechanism as `ChatMessageInteractiveFileNode`'s audio-transcription expand and the text/fact-check bubbles; the ListView clamps it to what fits.
429+
430+
## Rich-message media in the gallery / shared-media / preview pipelines (`Message.effectiveMedia`)
431+
432+
A rich message's media (images / videos / audio / documents) lives in `attribute.instantPage.media`, **not** in `message.media` (which is empty — rich messages are sent with `text: ""` and no media reference). To make that media participate in the *same* shared-media-index, gallery, file-list, playback, download, and save/copy pipelines that normal `message.media` flows through, there is one shared accessor and a set of opt-in call-site swaps.
433+
434+
### The accessor
435+
436+
`Message.effectiveMedia: [Media]` (+ a delegating `EngineMessage.effectiveMedia`) in `submodules/TelegramCore/Sources/Utils/MessageUtils.swift`:
437+
438+
```swift
439+
var effectiveMedia: [Media] {
440+
if !self.media.isEmpty { return self.media } // normal message: identical to message.media
441+
if let richText = self.richText { return richText.instantPage.allMedia() } // rich: the instant-page media
442+
return self.media
443+
}
444+
```
445+
446+
`Message.richText` (same file) is already a typed `RichTextMessageAttribute?`; `InstantPage.allMedia()` (`SyncCore_InstantPage.swift`) recursively gathers media from the page's blocks (audio/collage/cover/details/image/list/slideshow/video) via its `[MediaId: Media]` dict. **For a normal message `effectiveMedia == message.media`**, so swapping a `message.media` read for `message.effectiveMedia` is behavior-preserving for non-rich content and only adds the rich media where the site should consider it. **Scope is first-media** for now (call sites keep their `.first` / iterate-and-break logic; the helper returns all media but callers stop at the first match — the `//TODO:rewrite to take all media` markers remain).
447+
448+
### Where things live
449+
450+
| Layer | What |
451+
|---|---|
452+
| **Discovery / index** | `tagsForStoreMessage` (`StoreMessage_Telegram.swift`) indexes rich media into `MessageTags` (photo/video/gif/voice/file). **This is the linchpin**: it makes rich messages *appear* in every tag-queried surface (shared-media tabs, search, downloads) — which is exactly why each rendering-side site below then needs `effectiveMedia`, or it renders the surfaced message blank. |
453+
| **Extraction helper** | `Message.effectiveMedia` (above). |
454+
| **Shared-media grids / rows** | `PeerInfoVisualMediaPaneNode`, `PeerInfoGifPaneNode`, `ListMessageItem` (row-type selection) + `ListMessageFileItemNode` (file/music/voice row), `ChatListSearchMediaNode` (search media grid). |
455+
| **Gallery open + items** | `GalleryController` (`tagsForMessage` + `mediaForMessage` — the duplicated `message.media`/`message.richText` blocks were collapsed into one `effectiveMedia` loop), `GalleryData.chatMessageGalleryControllerData`, `SecretMediaPreviewController` (its own local `mediaForMessage`), and the gallery item nodes `ChatDocumentGalleryItem` / `ChatExternalFileGalleryItem` / `ChatAnimationGalleryItem` (these re-derive from `message.media` in `node()`, so a rich doc/animation rendered **blank** without the swap) + `UniversalVideoGalleryItem` secondary affordances + `ChatItemGalleryFooterContentNode`. |
456+
| **Playback** | `PeerMessagesMediaPlaylist.extractFileMedia` (the peer music/voice playlist), `OverlayAudioPlayerControllerNode` (audio context menu). |
457+
| **Resolution / downloads / cleanup** | `FetchedMediaResource.findMediaResourceById(message:)`, `SyncCore_RecentDownloadItem`, `StoreDownloadedMedia`, `DeleteMessages.addMessageMediaResourceIdsToRemove(message:)` (rich media was **leaking on delete**), `CollectCacheUsageStats`, `ChatHistoryListNode` (download manager), `ChatListSearch{ListPaneNode,ContainerNode}`. |
458+
| **Actions** | `ChatInterfaceStateContextMenus` (Save-to-Camera-Roll, copy-image, save-audio/music-to-files, debug/premium), `ChatControllerNode` (post-suggestion media ref), `ChatControllerLoadDisplayNode` (edit send-validation), `ShareController.saveToCameraRoll`. |
459+
460+
### Non-obvious invariants
461+
462+
- **The tag-index change is what creates the work.** `tagsForStoreMessage` surfacing rich messages into tag-queried lists, *without* the rendering-side `effectiveMedia` swaps, produces visible **blank cells / blank rows / wrong row types**. Index and render must move together.
463+
- **The rich message's own in-chat bubble + in-bubble gallery do NOT read `message.media`** — a rich message renders via `ChatMessageRichDataBubbleContentNode` (InstantPage V2), in-bubble image/video tap opens `InstantPageGalleryController` (reads the instant page directly), and in-bubble audio uses `InstantPageV2AudioContentNode`. So the text-bubble / interactive-file / interactive-media nodes' `message.media` reads are **never reached by a rich message** and are deliberately left alone.
464+
- **Do NOT route the FORWARD path through `effectiveMedia`** (`ChatControllerNode` `forwardedMessages` ~556/560/568). The `RichTextMessageAttribute` already travels with a forward, so the forwarded copy reconstructs from the attribute; injecting the instant-page media as top-level `message.media` there would **double-render** (rich bubble + a separate media attachment). That `message.media` processing is caption-hiding / poll-stripping only, both irrelevant to rich — left as `message.media`.
465+
- **Rich messages are edited as reconstructed MARKDOWN, not via the media-caption edit path.** So `ChatControllerLoadDisplayNode`'s edit caption-max-length / original-media-reference reads (~1241/1775/4463) stay on `message.media` — they belong to the `.media` edit state a rich message never enters. (The send-*validation* `.contains` at ~2273 IS swapped, so an edit that leaves only media isn't wrongly rejected.)
466+
- **`RichTextMessageAttribute.associatedMediaIds` stays `[]` — intentionally.** `MessageHistoryTable` resolves `associatedMediaIds` via `getMedia(id)` in the postbox **media table**, but rich-message media is embedded inside the attribute blob, not the table — so returning the keys would be a no-op without also inserting the media into the table. The embedded-blob approach is self-contained.
467+
- **`fullInstantPage` is not indexed** (the server doesn't index it either, and it's fetched on demand after store-time). The first media lives in the partial `instantPage` anyway.
468+
- **Only switch the loop SOURCE, never the per-type branches.** Many swapped loops still contain `TelegramMediaPoll`/`TelegramMediaPaidContent`/`TelegramMediaWebpage` branches that rich messages never match — that's fine and intentional; only the `for … in <msg>.media` source changes.
469+
- **Build-only completeness gate.** Every swap is type-identical (`[Media]` → `[Media]`), so the only compile risk is a receiver that is neither `Message` nor `EngineMessage`; the full Bazel build is the gate (no per-module build / unit tests). Deferred, NOT done: chat-list/reply/pinned/notification/forward thumbnail **previews** and the "Photo"/"Video" media-kind **labels** (`messageContentKind`/`ChatListItemStrings`) — those are preview surfaces, not blank-cell breakage — and **multi-media** (first-media-only is the current scope).

‎submodules/ChatListUI/Sources/ChatListSearchContainerNode.swift‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1416,7 +1416,7 @@ public final class ChatListSearchContainerNode: SearchDisplayControllerContentNo
14161416

14171417
var resourceIds = Set<EngineMediaResource.Id>()
14181418
for message in messages {
1419-
for media in message.media {
1419+
for media in message.effectiveMedia {
14201420
if let file = media as? TelegramMediaFile {
14211421
resourceIds.insert(EngineMediaResource.Id(file.resource.id))
14221422
}

‎submodules/ChatListUI/Sources/ChatListSearchListPaneNode.swift‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2106,7 +2106,7 @@ final class ChatListSearchListPaneNode: ASDisplayNode, ChatListSearchPaneNode {
21062106
let queryTokens = stringIndexTokens(query ?? "", transliteration: .combined)
21072107

21082108
func messageMatchesTokens(message: EngineMessage, tokens: [ValueBoxKey]) -> Bool {
2109-
for media in message.media {
2109+
for media in message.effectiveMedia {
21102110
if let file = media as? TelegramMediaFile {
21112111
if let fileName = file.fileName {
21122112
if matchStringIndexTokens(stringIndexTokens(fileName, transliteration: .none), with: tokens) {

‎submodules/ChatListUI/Sources/ChatListSearchMediaNode.swift‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -117,7 +117,7 @@ private final class VisualMediaItemNode: ASDisplayNode {
117117
if case .tap = gesture {
118118
if let _ = self.item {
119119
var media: EngineRawMedia?
120-
for value in message.media {
120+
for value in message.effectiveMedia {
121121
if let image = value as? TelegramMediaImage {
122122
media = image
123123
break
@@ -126,7 +126,7 @@ private final class VisualMediaItemNode: ASDisplayNode {
126126
break
127127
}
128128
}
129-
129+
130130
if let media = media {
131131
if let file = media as? TelegramMediaFile {
132132
if isMediaStreamable(message: EngineMessage(message), media: file) {
@@ -150,7 +150,7 @@ private final class VisualMediaItemNode: ASDisplayNode {
150150
}
151151

152152
var media: EngineRawMedia?
153-
for value in message.media {
153+
for value in message.effectiveMedia {
154154
if let image = value as? TelegramMediaImage {
155155
media = image
156156
break
@@ -159,7 +159,7 @@ private final class VisualMediaItemNode: ASDisplayNode {
159159
break
160160
}
161161
}
162-
162+
163163
if let resourceStatus = self.resourceStatus, let file = media as? TelegramMediaFile {
164164
switch resourceStatus {
165165
case .Fetching:
@@ -186,7 +186,7 @@ private final class VisualMediaItemNode: ASDisplayNode {
186186
self.theme = theme
187187
var media: EngineRawMedia?
188188
if let message = item.message {
189-
for value in message.media {
189+
for value in message.effectiveMedia {
190190
if let image = value as? TelegramMediaImage {
191191
media = image
192192
break
@@ -413,7 +413,7 @@ private final class VisualMediaItem {
413413

414414
var aspectRatio: CGFloat = 1.0
415415
var dimensions = CGSize(width: 100.0, height: 100.0)
416-
for media in message.media {
416+
for media in message.effectiveMedia {
417417
if let file = media as? TelegramMediaFile {
418418
if let dimensionsValue = file.dimensions, dimensions.height > 1 {
419419
dimensions = dimensionsValue.cgSize

‎submodules/GalleryData/Sources/GalleryData.swift‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -125,7 +125,7 @@ public func chatMessageGalleryControllerData(
125125
}
126126
}
127127
}
128-
for media in message.media {
128+
for media in message.effectiveMedia {
129129
if let poll = media as? TelegramMediaPoll {
130130
standalone = true
131131
galleryMedia = poll

‎submodules/GalleryUI/Sources/ChatItemGalleryFooterContentNode.swift‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -870,7 +870,7 @@ final class ChatItemGalleryFooterContentNode: GalleryFooterContentNode, ASScroll
870870
var canEdit = false
871871
var isImage = false
872872
var isVideo = false
873-
for media in message.media {
873+
for media in message.effectiveMedia {
874874
if media is TelegramMediaImage {
875875
canEdit = true
876876
isImage = true

‎submodules/GalleryUI/Sources/GalleryController.swift‎

Lines changed: 21 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -18,26 +18,27 @@ import UndoUI
1818
import TranslateUI
1919

2020
private func tagsForMessage(_ message: Message) -> MessageTags? {
21-
for media in message.media {
21+
//TODO:rewrite to take all media (effectiveMedia returns all rich-text media; we stop at the first)
22+
for media in message.effectiveMedia {
2223
switch media {
23-
case _ as TelegramMediaImage:
24-
return .photoOrVideo
25-
case let file as TelegramMediaFile:
26-
if file.isVideo {
27-
if file.isAnimated {
28-
return .gif
29-
} else {
30-
return .photoOrVideo
31-
}
32-
} else if file.isVoice {
33-
return .voiceOrInstantVideo
34-
} else if file.isSticker {
35-
return nil
24+
case _ as TelegramMediaImage:
25+
return .photoOrVideo
26+
case let file as TelegramMediaFile:
27+
if file.isVideo {
28+
if file.isAnimated {
29+
return .gif
3630
} else {
37-
return .file
31+
return .photoOrVideo
3832
}
39-
default:
40-
break
33+
} else if file.isVoice {
34+
return .voiceOrInstantVideo
35+
} else if file.isSticker {
36+
return nil
37+
} else {
38+
return .file
39+
}
40+
default:
41+
break
4142
}
4243
}
4344
return nil
@@ -61,7 +62,8 @@ private func galleryMediaForMedia(media: Media) -> Media? {
6162
}
6263

6364
func mediaForMessage(message: Message, mediaSubject: GalleryMediaSubject? = nil) -> [(Media, TelegramMediaImage?)] {
64-
for media in message.media {
65+
//TODO:rewrite to take all media (effectiveMedia returns all rich-text media; we return the first match)
66+
for media in message.effectiveMedia {
6567
if let result = galleryMediaForMedia(media: media) {
6668
return [(result, nil)]
6769
} else if let poll = media as? TelegramMediaPoll {
@@ -115,6 +117,7 @@ func mediaForMessage(message: Message, mediaSubject: GalleryMediaSubject? = nil)
115117
}
116118
}
117119
}
120+
118121
return []
119122
}
120123

‎submodules/GalleryUI/Sources/Items/ChatAnimationGalleryItem.swift‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ class ChatAnimationGalleryItem: GalleryItem {
3333
func node(synchronous: Bool) -> GalleryItemNode {
3434
let node = ChatAnimationGalleryItemNode(context: self.context, presentationData: self.presentationData)
3535

36-
for media in self.message.media {
36+
for media in self.message.effectiveMedia {
3737
if let file = media as? TelegramMediaFile {
3838
node.setFile(context: self.context, fileReference: .message(message: MessageReference(self.message), media: file))
3939
break

‎submodules/GalleryUI/Sources/Items/ChatDocumentGalleryItem.swift‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ class ChatDocumentGalleryItem: GalleryItem {
2929
func node(synchronous: Bool) -> GalleryItemNode {
3030
let node = ChatDocumentGalleryItemNode(context: self.context, presentationData: self.presentationData)
3131

32-
for media in self.message.media {
32+
for media in self.message.effectiveMedia {
3333
if let file = media as? TelegramMediaFile {
3434
node.setFile(context: context, fileReference: .message(message: MessageReference(self.message), media: file))
3535
break

0 commit comments

Comments
 (0)