[PATCH RFC 0/6] video: Add UTF-8 support for UEFI applications
Andre submitted 2 years ago DM_VIDEO improvements for UEFI applications using UEFI's EFI_SIMPLE_TEXT_OUTPUT_PROTOCOL but did not follow with suggested changes. This series takes care of the UTF-8 support which required to draw symbol and box drawing characters used by UEFI applications like grub2 and sd-boot correctly.
Compared to Andre's version this version has the following changes: - use and extend existing conversion functions from lib/charset.c - convert first to UTF-32 to support the truetype console as well - conversion is conditional on CONFIG_EFI_LOADER - use escape sequences in tests as proposed by Heinrich
Link: https://lore.kernel.org/u-boot/20220110005638.21599-1-andre.przywara@arm.com... Signed-off-by: Janne Grunau j@jannau.net --- Andre Przywara (2): efi_selftest: Add international characters test efi_selftest: Add box drawing character selftest
Janne Grunau (4): lib: charset: Fix utf8_to_utf32_stream() return value doc string video: console: Parse UTF-8 character sequences lib/charset: Map cp437 low chars (0x01 - 0x1f) from unicode efi_selftest: Add symbol character selftest
drivers/video/console_normal.c | 6 +++-- drivers/video/console_rotate.c | 16 +++++++++----- drivers/video/console_truetype.c | 8 +++---- drivers/video/vidconsole-uclass.c | 18 ++++++++++----- drivers/video/vidconsole_internal.h | 15 +++++++++++++ include/charset.h | 4 ++-- include/cp1250.h | 12 ++++++++-- include/cp437.h | 12 ++++++++-- include/video_console.h | 10 +++++---- lib/charset.c | 9 +++++--- lib/efi_loader/efi_unicode_collation.c | 2 +- lib/efi_selftest/efi_selftest_textoutput.c | 35 ++++++++++++++++++++++++++++++ 12 files changed, 116 insertions(+), 31 deletions(-) --- base-commit: 866ca972d6c3cabeaf6dbac431e8e08bb30b3c8e change-id: 20240117-vidconsole-utf8-uefi-fa23b4ac65d6
Best regards,
From: Janne Grunau j@jannau.net
The comment appears to be copied from utf8_to_cp437_stream() but was not updated.
Signed-off-by: Janne Grunau j@jannau.net --- include/charset.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/include/charset.h b/include/charset.h index 44034c71d3..714382e1c1 100644 --- a/include/charset.h +++ b/include/charset.h @@ -328,7 +328,7 @@ int utf8_to_cp437_stream(u8 c, char *buffer); * * @c: next UTF-8 character to convert * @buffer: buffer, at least 5 characters - * Return: next codepage 437 character or 0 + * Return: next Unicode code point or 0 */ int utf8_to_utf32_stream(u8 c, char *buffer);
On Wed, 17 Jan 2024 23:24:27 +0100 Janne Grunau via B4 Relay devnull+j.jannau.net@kernel.org wrote:
From: Janne Grunau j@jannau.net
The comment appears to be copied from utf8_to_cp437_stream() but was not updated.
Signed-off-by: Janne Grunau j@jannau.net
Reviewed-by: Andre Przywara andre.przywara@arm.com
Cheers, Andre
include/charset.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/include/charset.h b/include/charset.h index 44034c71d3..714382e1c1 100644 --- a/include/charset.h +++ b/include/charset.h @@ -328,7 +328,7 @@ int utf8_to_cp437_stream(u8 c, char *buffer);
- @c: next UTF-8 character to convert
- @buffer: buffer, at least 5 characters
- Return: next codepage 437 character or 0
*/
- Return: next Unicode code point or 0
int utf8_to_utf32_stream(u8 c, char *buffer);
On 1/17/24 23:24, Janne Grunau via B4 Relay wrote:
From: Janne Grunau j@jannau.net
The comment appears to be copied from utf8_to_cp437_stream() but was not updated.
Fixes: e91789e2f661 ("lib/charset: UTF-8 stream conversion")
Signed-off-by: Janne Grunau j@jannau.net
include/charset.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/include/charset.h b/include/charset.h index 44034c71d3..714382e1c1 100644 --- a/include/charset.h +++ b/include/charset.h @@ -328,7 +328,7 @@ int utf8_to_cp437_stream(u8 c, char *buffer);
Without a further description it is unclear what this function is intended to do. How about:
"This function can be used to convert a UTF-8 byte-stream to Unicode code-points.
The function is called for each byte @c in a UTF-8 stream. The byte is appended to the temporary storage @buffer until the UTF-8 stream in @buffer describes a Unicode code point.
When a new code point has been decoded it is returned and buffer[0] is set to '\0', otherwise the return value is 0.
The buffer must be at least 5 characters long. Before the first function invocation buffer[0] must be set to '\0'."
- @c: next UTF-8 character to convert
- @buffer: buffer, at least 5 characters
- Return: next codepage 437 character or 0
- Return: next Unicode code point or 0
%s/next//g
*/ int utf8_to_utf32_stream(u8 c, char *buffer);
Best regards
Heinrich
From: Janne Grunau j@jannau.net
efi_console / UEFI applications (grub2, sd-boot, ...) pass UTF-8 character sequences to vidconsole which results in wrong glyphs for code points outside of ASCII. The truetype console expects Unicode code points and bitmap font based consoles expect code page 437 code points. To support both convert UTF-8 to UTF-32 and pass Unicode code points in vidconsole_ops.putc_xy(). These can be used directly in console_truetype and after conversion to code page 437 in console_{normal,rotate}.
This fixes rendering of international, symbol and box drawing characters used by UEFI applications.
Signed-off-by: Janne Grunau j@jannau.net --- drivers/video/console_normal.c | 6 ++++-- drivers/video/console_rotate.c | 16 ++++++++++------ drivers/video/console_truetype.c | 8 ++++---- drivers/video/vidconsole-uclass.c | 18 +++++++++++++----- drivers/video/vidconsole_internal.h | 15 +++++++++++++++ include/video_console.h | 10 ++++++---- 6 files changed, 52 insertions(+), 21 deletions(-)
diff --git a/drivers/video/console_normal.c b/drivers/video/console_normal.c index a0231293f3..34ef5a5229 100644 --- a/drivers/video/console_normal.c +++ b/drivers/video/console_normal.c @@ -7,6 +7,7 @@ */
#include <common.h> +#include <charset.h> #include <dm.h> #include <video.h> #include <video_console.h> @@ -63,7 +64,7 @@ static int console_move_rows(struct udevice *dev, uint rowdst, return 0; }
-static int console_putc_xy(struct udevice *dev, uint x_frac, uint y, char ch) +static int console_putc_xy(struct udevice *dev, uint x_frac, uint y, int cp) { struct vidconsole_priv *vc_priv = dev_get_uclass_priv(dev); struct udevice *vid = dev->parent; @@ -73,8 +74,9 @@ static int console_putc_xy(struct udevice *dev, uint x_frac, uint y, char ch) int pbytes = VNBYTES(vid_priv->bpix); int x, linenum, ret; void *start, *line; + u8 ch = console_utf_to_cp437(cp); uchar *pfont = fontdata->video_fontdata + - (u8)ch * fontdata->char_pixel_bytes; + ch * fontdata->char_pixel_bytes;
if (x_frac + VID_TO_POS(vc_priv->x_charsize) > vc_priv->xsize_frac) return -EAGAIN; diff --git a/drivers/video/console_rotate.c b/drivers/video/console_rotate.c index 65358a1c6e..e4303dfb36 100644 --- a/drivers/video/console_rotate.c +++ b/drivers/video/console_rotate.c @@ -7,6 +7,7 @@ */
#include <common.h> +#include <charset.h> #include <dm.h> #include <video.h> #include <video_console.h> @@ -67,7 +68,7 @@ static int console_move_rows_1(struct udevice *dev, uint rowdst, uint rowsrc, return 0; }
-static int console_putc_xy_1(struct udevice *dev, uint x_frac, uint y, char ch) +static int console_putc_xy_1(struct udevice *dev, uint x_frac, uint y, int cp) { struct vidconsole_priv *vc_priv = dev_get_uclass_priv(dev); struct udevice *vid = dev->parent; @@ -77,8 +78,9 @@ static int console_putc_xy_1(struct udevice *dev, uint x_frac, uint y, char ch) int pbytes = VNBYTES(vid_priv->bpix); int x, linenum, ret; void *start, *line; + u8 ch = console_utf_to_cp437(cp); uchar *pfont = fontdata->video_fontdata + - (u8)ch * fontdata->char_pixel_bytes; + ch * fontdata->char_pixel_bytes;
if (x_frac + VID_TO_POS(vc_priv->x_charsize) > vc_priv->xsize_frac) return -EAGAIN; @@ -145,7 +147,7 @@ static int console_move_rows_2(struct udevice *dev, uint rowdst, uint rowsrc, return 0; }
-static int console_putc_xy_2(struct udevice *dev, uint x_frac, uint y, char ch) +static int console_putc_xy_2(struct udevice *dev, uint x_frac, uint y, int cp) { struct vidconsole_priv *vc_priv = dev_get_uclass_priv(dev); struct udevice *vid = dev->parent; @@ -155,8 +157,9 @@ static int console_putc_xy_2(struct udevice *dev, uint x_frac, uint y, char ch) int pbytes = VNBYTES(vid_priv->bpix); int linenum, x, ret; void *start, *line; + u8 ch = console_utf_to_cp437(cp); uchar *pfont = fontdata->video_fontdata + - (u8)ch * fontdata->char_pixel_bytes; + ch * fontdata->char_pixel_bytes;
if (x_frac + VID_TO_POS(vc_priv->x_charsize) > vc_priv->xsize_frac) return -EAGAIN; @@ -227,7 +230,7 @@ static int console_move_rows_3(struct udevice *dev, uint rowdst, uint rowsrc, return 0; }
-static int console_putc_xy_3(struct udevice *dev, uint x_frac, uint y, char ch) +static int console_putc_xy_3(struct udevice *dev, uint x_frac, uint y, int cp) { struct vidconsole_priv *vc_priv = dev_get_uclass_priv(dev); struct udevice *vid = dev->parent; @@ -237,8 +240,9 @@ static int console_putc_xy_3(struct udevice *dev, uint x_frac, uint y, char ch) int pbytes = VNBYTES(vid_priv->bpix); int linenum, x, ret; void *start, *line; + u8 ch = console_utf_to_cp437(cp); uchar *pfont = fontdata->video_fontdata + - (u8)ch * fontdata->char_pixel_bytes; + ch * fontdata->char_pixel_bytes;
if (x_frac + VID_TO_POS(vc_priv->x_charsize) > vc_priv->xsize_frac) return -EAGAIN; diff --git a/drivers/video/console_truetype.c b/drivers/video/console_truetype.c index 14fb81e956..bc3ec1c31f 100644 --- a/drivers/video/console_truetype.c +++ b/drivers/video/console_truetype.c @@ -262,7 +262,7 @@ static int console_truetype_move_rows(struct udevice *dev, uint rowdst, }
static int console_truetype_putc_xy(struct udevice *dev, uint x, uint y, - char ch) + int cp) { struct vidconsole_priv *vc_priv = dev_get_uclass_priv(dev); struct udevice *vid = dev->parent; @@ -281,7 +281,7 @@ static int console_truetype_putc_xy(struct udevice *dev, uint x, uint y, int row, ret;
/* First get some basic metrics about this character */ - stbtt_GetCodepointHMetrics(font, ch, &advance, &lsb); + stbtt_GetCodepointHMetrics(font, cp, &advance, &lsb);
/* * First out our current X position in fractional pixels. If we wrote @@ -290,7 +290,7 @@ static int console_truetype_putc_xy(struct udevice *dev, uint x, uint y, xpos = frac(VID_TO_PIXEL((double)x)); if (vc_priv->last_ch) { xpos += met->scale * stbtt_GetCodepointKernAdvance(font, - vc_priv->last_ch, ch); + vc_priv->last_ch, cp); }
/* @@ -320,7 +320,7 @@ static int console_truetype_putc_xy(struct udevice *dev, uint x, uint y, * return NULL; */ data = stbtt_GetCodepointBitmapSubpixel(font, met->scale, met->scale, - x_shift, 0, ch, &width, &height, + x_shift, 0, cp, &width, &height, &xoff, &yoff); if (!data) return width_frac; diff --git a/drivers/video/vidconsole-uclass.c b/drivers/video/vidconsole-uclass.c index 22d55df71f..ce1668b97b 100644 --- a/drivers/video/vidconsole-uclass.c +++ b/drivers/video/vidconsole-uclass.c @@ -11,6 +11,7 @@
#include <common.h> #include <abuf.h> +#include <charset.h> #include <command.h> #include <console.h> #include <log.h> @@ -20,7 +21,7 @@ #include <video_font.h> /* Bitmap font for code page 437 */ #include <linux/ctype.h>
-int vidconsole_putc_xy(struct udevice *dev, uint x, uint y, char ch) +int vidconsole_putc_xy(struct udevice *dev, uint x, uint y, int ch) { struct vidconsole_ops *ops = vidconsole_get_ops(dev);
@@ -426,8 +427,8 @@ error: priv->escape = 0; }
-/* Put that actual character on the screen (using the CP437 code page). */ -static int vidconsole_output_glyph(struct udevice *dev, char ch) +/* Put that actual character on the screen (using the UTF-32 code points). */ +static int vidconsole_output_glyph(struct udevice *dev, int ch) { struct vidconsole_priv *priv = dev_get_uclass_priv(dev); int ret; @@ -455,7 +456,7 @@ static int vidconsole_output_glyph(struct udevice *dev, char ch) int vidconsole_put_char(struct udevice *dev, char ch) { struct vidconsole_priv *priv = dev_get_uclass_priv(dev); - int ret; + int cp, ret;
if (priv->escape) { vidconsole_escape_char(dev, ch); @@ -489,7 +490,14 @@ int vidconsole_put_char(struct udevice *dev, char ch) priv->last_ch = 0; break; default: - ret = vidconsole_output_glyph(dev, ch); + if (CONFIG_EFI_LOADER) { + cp = utf8_to_utf32_stream(ch, priv->utf8_buf); + if (cp == 0) + return 0; + } else { + cp = ch; + } + ret = vidconsole_output_glyph(dev, cp); if (ret < 0) return ret; break; diff --git a/drivers/video/vidconsole_internal.h b/drivers/video/vidconsole_internal.h index 0ec581b266..3d19410a52 100644 --- a/drivers/video/vidconsole_internal.h +++ b/drivers/video/vidconsole_internal.h @@ -6,6 +6,9 @@ * (C) Copyright 2023 Dzmitry Sankouski dsankouski@gmail.com */
+#include <charset.h> +#include <config.h> + #define FLIPPED_DIRECTION 1 #define NORMAL_DIRECTION 0
@@ -142,3 +145,15 @@ int console_simple_get_font(struct udevice *dev, int seq, struct vidfont_info *i * See details in video_console.h select_font function **/ int console_simple_select_font(struct udevice *dev, const char *name, uint size); + +/** + * + */ +static inline u8 console_utf_to_cp437(int codepoint) +{ + if (CONFIG_EFI_LOADER) { + utf_to_cp(&codepoint, codepage_437); + return (u8)codepoint; + } + return (u8)codepoint; +} diff --git a/include/video_console.h b/include/video_console.h index bde67fa9a5..20cf2d0101 100644 --- a/include/video_console.h +++ b/include/video_console.h @@ -43,6 +43,7 @@ enum { * @col_saved: Saved X position, in fractional units (VID_TO_POS(x)) * @row_saved: Saved Y position in pixels (0=top) * @escape_buf: Buffer to accumulate escape sequence + * @utf8_buf: Buffer to accumulate UTF-8 byte sequence */ struct vidconsole_priv { struct stdio_dev sdev; @@ -66,6 +67,7 @@ struct vidconsole_priv { int row_saved; int col_saved; char escape_buf[32]; + char utf8_buf[8]; };
/** @@ -124,12 +126,12 @@ struct vidconsole_ops { * @x_frac: Fractional pixel X position (0=left-most pixel) which * is the X position multipled by VID_FRAC_DIV. * @y: Pixel Y position (0=top-most pixel) - * @ch: Character to write + * @cp: UTF-32 code point to write * @return number of fractional pixels that the cursor should move, * if all is OK, -EAGAIN if we ran out of space on this line, other -ve * on error */ - int (*putc_xy)(struct udevice *dev, uint x_frac, uint y, char ch); + int (*putc_xy)(struct udevice *dev, uint x_frac, uint y, int cp);
/** * move_rows() - Move text rows from one place to another @@ -403,12 +405,12 @@ void vidconsole_pop_colour(struct udevice *dev, struct vidconsole_colour *old); * @x_frac: Fractional pixel X position (0=left-most pixel) which * is the X position multipled by VID_FRAC_DIV. * @y: Pixel Y position (0=top-most pixel) - * @ch: Character to write + * @cp: UTF-32 code point to write * Return: number of fractional pixels that the cursor should move, * if all is OK, -EAGAIN if we ran out of space on this line, other -ve * on error */ -int vidconsole_putc_xy(struct udevice *dev, uint x, uint y, char ch); +int vidconsole_putc_xy(struct udevice *dev, uint x, uint y, int cp);
/** * vidconsole_move_rows() - Move text rows from one place to another
On Wed, 17 Jan 2024 23:24:28 +0100 Janne Grunau via B4 Relay devnull+j.jannau.net@kernel.org wrote:
Hi,
From: Janne Grunau j@jannau.net
efi_console / UEFI applications (grub2, sd-boot, ...) pass UTF-8 character sequences to vidconsole which results in wrong glyphs for code points outside of ASCII. The truetype console expects Unicode code points and bitmap font based consoles expect code page 437 code points. To support both convert UTF-8 to UTF-32 and pass Unicode code points in vidconsole_ops.putc_xy(). These can be used directly in console_truetype and after conversion to code page 437 in console_{normal,rotate}.
This fixes rendering of international, symbol and box drawing characters used by UEFI applications.
Signed-off-by: Janne Grunau j@jannau.net
drivers/video/console_normal.c | 6 ++++-- drivers/video/console_rotate.c | 16 ++++++++++------ drivers/video/console_truetype.c | 8 ++++---- drivers/video/vidconsole-uclass.c | 18 +++++++++++++----- drivers/video/vidconsole_internal.h | 15 +++++++++++++++ include/video_console.h | 10 ++++++---- 6 files changed, 52 insertions(+), 21 deletions(-)
diff --git a/drivers/video/console_normal.c b/drivers/video/console_normal.c index a0231293f3..34ef5a5229 100644 --- a/drivers/video/console_normal.c +++ b/drivers/video/console_normal.c @@ -7,6 +7,7 @@ */
#include <common.h> +#include <charset.h> #include <dm.h> #include <video.h> #include <video_console.h> @@ -63,7 +64,7 @@ static int console_move_rows(struct udevice *dev, uint rowdst, return 0; }
-static int console_putc_xy(struct udevice *dev, uint x_frac, uint y, char ch) +static int console_putc_xy(struct udevice *dev, uint x_frac, uint y, int cp) { struct vidconsole_priv *vc_priv = dev_get_uclass_priv(dev); struct udevice *vid = dev->parent; @@ -73,8 +74,9 @@ static int console_putc_xy(struct udevice *dev, uint x_frac, uint y, char ch) int pbytes = VNBYTES(vid_priv->bpix); int x, linenum, ret; void *start, *line;
- u8 ch = console_utf_to_cp437(cp); uchar *pfont = fontdata->video_fontdata +
(u8)ch * fontdata->char_pixel_bytes;
ch * fontdata->char_pixel_bytes;if (x_frac + VID_TO_POS(vc_priv->x_charsize) > vc_priv->xsize_frac) return -EAGAIN;
diff --git a/drivers/video/console_rotate.c b/drivers/video/console_rotate.c index 65358a1c6e..e4303dfb36 100644 --- a/drivers/video/console_rotate.c +++ b/drivers/video/console_rotate.c @@ -7,6 +7,7 @@ */
#include <common.h> +#include <charset.h> #include <dm.h> #include <video.h> #include <video_console.h> @@ -67,7 +68,7 @@ static int console_move_rows_1(struct udevice *dev, uint rowdst, uint rowsrc, return 0; }
-static int console_putc_xy_1(struct udevice *dev, uint x_frac, uint y, char ch) +static int console_putc_xy_1(struct udevice *dev, uint x_frac, uint y, int cp) { struct vidconsole_priv *vc_priv = dev_get_uclass_priv(dev); struct udevice *vid = dev->parent; @@ -77,8 +78,9 @@ static int console_putc_xy_1(struct udevice *dev, uint x_frac, uint y, char ch) int pbytes = VNBYTES(vid_priv->bpix); int x, linenum, ret; void *start, *line;
- u8 ch = console_utf_to_cp437(cp); uchar *pfont = fontdata->video_fontdata +
(u8)ch * fontdata->char_pixel_bytes;
ch * fontdata->char_pixel_bytes;if (x_frac + VID_TO_POS(vc_priv->x_charsize) > vc_priv->xsize_frac) return -EAGAIN;
@@ -145,7 +147,7 @@ static int console_move_rows_2(struct udevice *dev, uint rowdst, uint rowsrc, return 0; }
-static int console_putc_xy_2(struct udevice *dev, uint x_frac, uint y, char ch) +static int console_putc_xy_2(struct udevice *dev, uint x_frac, uint y, int cp) { struct vidconsole_priv *vc_priv = dev_get_uclass_priv(dev); struct udevice *vid = dev->parent; @@ -155,8 +157,9 @@ static int console_putc_xy_2(struct udevice *dev, uint x_frac, uint y, char ch) int pbytes = VNBYTES(vid_priv->bpix); int linenum, x, ret; void *start, *line;
- u8 ch = console_utf_to_cp437(cp); uchar *pfont = fontdata->video_fontdata +
(u8)ch * fontdata->char_pixel_bytes;
ch * fontdata->char_pixel_bytes;if (x_frac + VID_TO_POS(vc_priv->x_charsize) > vc_priv->xsize_frac) return -EAGAIN;
@@ -227,7 +230,7 @@ static int console_move_rows_3(struct udevice *dev, uint rowdst, uint rowsrc, return 0; }
-static int console_putc_xy_3(struct udevice *dev, uint x_frac, uint y, char ch) +static int console_putc_xy_3(struct udevice *dev, uint x_frac, uint y, int cp) { struct vidconsole_priv *vc_priv = dev_get_uclass_priv(dev); struct udevice *vid = dev->parent; @@ -237,8 +240,9 @@ static int console_putc_xy_3(struct udevice *dev, uint x_frac, uint y, char ch) int pbytes = VNBYTES(vid_priv->bpix); int linenum, x, ret; void *start, *line;
- u8 ch = console_utf_to_cp437(cp); uchar *pfont = fontdata->video_fontdata +
(u8)ch * fontdata->char_pixel_bytes;
ch * fontdata->char_pixel_bytes;if (x_frac + VID_TO_POS(vc_priv->x_charsize) > vc_priv->xsize_frac) return -EAGAIN;
diff --git a/drivers/video/console_truetype.c b/drivers/video/console_truetype.c index 14fb81e956..bc3ec1c31f 100644 --- a/drivers/video/console_truetype.c +++ b/drivers/video/console_truetype.c @@ -262,7 +262,7 @@ static int console_truetype_move_rows(struct udevice *dev, uint rowdst, }
static int console_truetype_putc_xy(struct udevice *dev, uint x, uint y,
char ch)
int cp){ struct vidconsole_priv *vc_priv = dev_get_uclass_priv(dev); struct udevice *vid = dev->parent; @@ -281,7 +281,7 @@ static int console_truetype_putc_xy(struct udevice *dev, uint x, uint y, int row, ret;
/* First get some basic metrics about this character */
- stbtt_GetCodepointHMetrics(font, ch, &advance, &lsb);
stbtt_GetCodepointHMetrics(font, cp, &advance, &lsb);
/*
- First out our current X position in fractional pixels. If we wrote
@@ -290,7 +290,7 @@ static int console_truetype_putc_xy(struct udevice *dev, uint x, uint y, xpos = frac(VID_TO_PIXEL((double)x)); if (vc_priv->last_ch) { xpos += met->scale * stbtt_GetCodepointKernAdvance(font,
vc_priv->last_ch, ch);
vc_priv->last_ch, cp);}
/*
@@ -320,7 +320,7 @@ static int console_truetype_putc_xy(struct udevice *dev, uint x, uint y, * return NULL; */ data = stbtt_GetCodepointBitmapSubpixel(font, met->scale, met->scale,
x_shift, 0, ch, &width, &height,
x_shift, 0, cp, &width, &height, &xoff, &yoff);diff --git a/drivers/video/vidconsole-uclass.c b/drivers/video/vidconsole-uclass.c index 22d55df71f..ce1668b97b 100644 --- a/drivers/video/vidconsole-uclass.c +++ b/drivers/video/vidconsole-uclass.c @@ -11,6 +11,7 @@
#include <common.h> #include <abuf.h> +#include <charset.h> #include <command.h> #include <console.h> #include <log.h> @@ -20,7 +21,7 @@ #include <video_font.h> /* Bitmap font for code page 437 */ #include <linux/ctype.h>
-int vidconsole_putc_xy(struct udevice *dev, uint x, uint y, char ch) +int vidconsole_putc_xy(struct udevice *dev, uint x, uint y, int ch) { struct vidconsole_ops *ops = vidconsole_get_ops(dev);
@@ -426,8 +427,8 @@ error: priv->escape = 0; }
-/* Put that actual character on the screen (using the CP437 code page). */ -static int vidconsole_output_glyph(struct udevice *dev, char ch) +/* Put that actual character on the screen (using the UTF-32 code points). */ +static int vidconsole_output_glyph(struct udevice *dev, int ch) { struct vidconsole_priv *priv = dev_get_uclass_priv(dev); int ret; @@ -455,7 +456,7 @@ static int vidconsole_output_glyph(struct udevice *dev, char ch) int vidconsole_put_char(struct udevice *dev, char ch)
This looks like a weird interface, to stuff in 8-bit bytes really, rather than the promised characters, and have some kind of internal state here. I am wondering if we should make this function static? There is only one real user (in board/atmel/common/video_display.c), that looks like it could use vidconsole_put_string(). Plus the tests, but we could fix them as well. Or we make this function also take a UTF-32?
Shouldn't hold back this series, but just stumbled upon it, and was curious what other people think.
{ struct vidconsole_priv *priv = dev_get_uclass_priv(dev);
- int ret;
int cp, ret;
if (priv->escape) { vidconsole_escape_char(dev, ch);
@@ -489,7 +490,14 @@ int vidconsole_put_char(struct udevice *dev, char ch) priv->last_ch = 0; break; default:
ret = vidconsole_output_glyph(dev, ch);
if (CONFIG_EFI_LOADER) {
This would need to be: if (IS_ENABLED(CONFIG_EFI_LOADER)) { Otherwise disabling EFI_LOADER breaks compilation, since CONFIG_EFI_LOADER will not be defined at all:
drivers/video/vidconsole-uclass.c: In function ‘vidconsole_put_char’: drivers/video/vidconsole-uclass.c:493:7: error: ‘CONFIG_EFI_LOADER’ undeclared (first use in this function); did you mean ‘CONFIG_ENV_ADDR’?
But more general, is this really a compile time check? Without EFI applications, do we only expect 7-bit ASCII, or CP437 explicitly? It's only U-Boot generated output then, right?
cp = utf8_to_utf32_stream(ch, priv->utf8_buf);if (cp == 0)return 0;} else {cp = ch;}ret = vidconsole_output_glyph(dev, cp);diff --git a/drivers/video/vidconsole_internal.h b/drivers/video/vidconsole_internal.h index 0ec581b266..3d19410a52 100644 --- a/drivers/video/vidconsole_internal.h +++ b/drivers/video/vidconsole_internal.h @@ -6,6 +6,9 @@
- (C) Copyright 2023 Dzmitry Sankouski dsankouski@gmail.com
*/
+#include <charset.h> +#include <config.h>
#define FLIPPED_DIRECTION 1 #define NORMAL_DIRECTION 0
@@ -142,3 +145,15 @@ int console_simple_get_font(struct udevice *dev, int seq, struct vidfont_info *i
- See details in video_console.h select_font function
**/ int console_simple_select_font(struct udevice *dev, const char *name, uint size);
+/**
- */
+static inline u8 console_utf_to_cp437(int codepoint) +{
- if (CONFIG_EFI_LOADER) {
Same IS_ENABLED() usage required here.
Cheers, Andre.
utf_to_cp(&codepoint, codepage_437);return (u8)codepoint;- }
- return (u8)codepoint;
+} diff --git a/include/video_console.h b/include/video_console.h index bde67fa9a5..20cf2d0101 100644 --- a/include/video_console.h +++ b/include/video_console.h @@ -43,6 +43,7 @@ enum {
- @col_saved: Saved X position, in fractional units (VID_TO_POS(x))
- @row_saved: Saved Y position in pixels (0=top)
- @escape_buf: Buffer to accumulate escape sequence
*/
- @utf8_buf: Buffer to accumulate UTF-8 byte sequence
struct vidconsole_priv { struct stdio_dev sdev; @@ -66,6 +67,7 @@ struct vidconsole_priv { int row_saved; int col_saved; char escape_buf[32];
- char utf8_buf[8];
};
/** @@ -124,12 +126,12 @@ struct vidconsole_ops { * @x_frac: Fractional pixel X position (0=left-most pixel) which * is the X position multipled by VID_FRAC_DIV. * @y: Pixel Y position (0=top-most pixel)
* @ch: Character to write
* @cp: EF UTF-32 code point to write*/
- @return number of fractional pixels that the cursor should move,
- if all is OK, -EAGAIN if we ran out of space on this line, other -ve
- on error
- int (*putc_xy)(struct udevice *dev, uint x_frac, uint y, char ch);
int (*putc_xy)(struct udevice *dev, uint x_frac, uint y, int cp);
/**
- move_rows() - Move text rows from one place to another
@@ -403,12 +405,12 @@ void vidconsole_pop_colour(struct udevice *dev, struct vidconsole_colour *old);
- @x_frac: Fractional pixel X position (0=left-most pixel) which
is the X position multipled by VID_FRAC_DIV.- @y: Pixel Y position (0=top-most pixel)
- @ch: Character to write
*/
- @cp: UTF-32 code point to write
- Return: number of fractional pixels that the cursor should move,
- if all is OK, -EAGAIN if we ran out of space on this line, other -ve
- on error
-int vidconsole_putc_xy(struct udevice *dev, uint x, uint y, char ch); +int vidconsole_putc_xy(struct udevice *dev, uint x, uint y, int cp);
/**
- vidconsole_move_rows() - Move text rows from one place to another
On 1/18/24 14:57, Andre Przywara wrote:
On Wed, 17 Jan 2024 23:24:28 +0100 Janne Grunau via B4 Relay devnull+j.jannau.net@kernel.org wrote:
Hi,
From: Janne Grunau j@jannau.net
efi_console / UEFI applications (grub2, sd-boot, ...) pass UTF-8 character sequences to vidconsole which results in wrong glyphs for code points outside of ASCII. The truetype console expects Unicode code points and bitmap font based consoles expect code page 437 code points. To support both convert UTF-8 to UTF-32 and pass Unicode code points in vidconsole_ops.putc_xy(). These can be used directly in console_truetype and after conversion to code page 437 in console_{normal,rotate}.
This fixes rendering of international, symbol and box drawing characters used by UEFI applications.
Signed-off-by: Janne Grunau j@jannau.net
drivers/video/console_normal.c | 6 ++++-- drivers/video/console_rotate.c | 16 ++++++++++------ drivers/video/console_truetype.c | 8 ++++---- drivers/video/vidconsole-uclass.c | 18 +++++++++++++----- drivers/video/vidconsole_internal.h | 15 +++++++++++++++ include/video_console.h | 10 ++++++---- 6 files changed, 52 insertions(+), 21 deletions(-)
diff --git a/drivers/video/console_normal.c b/drivers/video/console_normal.c index a0231293f3..34ef5a5229 100644 --- a/drivers/video/console_normal.c +++ b/drivers/video/console_normal.c @@ -7,6 +7,7 @@ */
#include <common.h> +#include <charset.h> #include <dm.h> #include <video.h> #include <video_console.h> @@ -63,7 +64,7 @@ static int console_move_rows(struct udevice *dev, uint rowdst, return 0; }
-static int console_putc_xy(struct udevice *dev, uint x_frac, uint y, char ch) +static int console_putc_xy(struct udevice *dev, uint x_frac, uint y, int cp) { struct vidconsole_priv *vc_priv = dev_get_uclass_priv(dev); struct udevice *vid = dev->parent; @@ -73,8 +74,9 @@ static int console_putc_xy(struct udevice *dev, uint x_frac, uint y, char ch) int pbytes = VNBYTES(vid_priv->bpix); int x, linenum, ret; void *start, *line;
- u8 ch = console_utf_to_cp437(cp); uchar *pfont = fontdata->video_fontdata +
(u8)ch * fontdata->char_pixel_bytes;
ch * fontdata->char_pixel_bytes;if (x_frac + VID_TO_POS(vc_priv->x_charsize) > vc_priv->xsize_frac) return -EAGAIN;
diff --git a/drivers/video/console_rotate.c b/drivers/video/console_rotate.c index 65358a1c6e..e4303dfb36 100644 --- a/drivers/video/console_rotate.c +++ b/drivers/video/console_rotate.c @@ -7,6 +7,7 @@ */
#include <common.h> +#include <charset.h> #include <dm.h> #include <video.h> #include <video_console.h> @@ -67,7 +68,7 @@ static int console_move_rows_1(struct udevice *dev, uint rowdst, uint rowsrc, return 0; }
-static int console_putc_xy_1(struct udevice *dev, uint x_frac, uint y, char ch) +static int console_putc_xy_1(struct udevice *dev, uint x_frac, uint y, int cp) { struct vidconsole_priv *vc_priv = dev_get_uclass_priv(dev); struct udevice *vid = dev->parent; @@ -77,8 +78,9 @@ static int console_putc_xy_1(struct udevice *dev, uint x_frac, uint y, char ch) int pbytes = VNBYTES(vid_priv->bpix); int x, linenum, ret; void *start, *line;
- u8 ch = console_utf_to_cp437(cp); uchar *pfont = fontdata->video_fontdata +
(u8)ch * fontdata->char_pixel_bytes;
ch * fontdata->char_pixel_bytes;if (x_frac + VID_TO_POS(vc_priv->x_charsize) > vc_priv->xsize_frac) return -EAGAIN;
@@ -145,7 +147,7 @@ static int console_move_rows_2(struct udevice *dev, uint rowdst, uint rowsrc, return 0; }
-static int console_putc_xy_2(struct udevice *dev, uint x_frac, uint y, char ch) +static int console_putc_xy_2(struct udevice *dev, uint x_frac, uint y, int cp) { struct vidconsole_priv *vc_priv = dev_get_uclass_priv(dev); struct udevice *vid = dev->parent; @@ -155,8 +157,9 @@ static int console_putc_xy_2(struct udevice *dev, uint x_frac, uint y, char ch) int pbytes = VNBYTES(vid_priv->bpix); int linenum, x, ret; void *start, *line;
- u8 ch = console_utf_to_cp437(cp); uchar *pfont = fontdata->video_fontdata +
(u8)ch * fontdata->char_pixel_bytes;
ch * fontdata->char_pixel_bytes;if (x_frac + VID_TO_POS(vc_priv->x_charsize) > vc_priv->xsize_frac) return -EAGAIN;
@@ -227,7 +230,7 @@ static int console_move_rows_3(struct udevice *dev, uint rowdst, uint rowsrc, return 0; }
-static int console_putc_xy_3(struct udevice *dev, uint x_frac, uint y, char ch) +static int console_putc_xy_3(struct udevice *dev, uint x_frac, uint y, int cp) { struct vidconsole_priv *vc_priv = dev_get_uclass_priv(dev); struct udevice *vid = dev->parent; @@ -237,8 +240,9 @@ static int console_putc_xy_3(struct udevice *dev, uint x_frac, uint y, char ch) int pbytes = VNBYTES(vid_priv->bpix); int linenum, x, ret; void *start, *line;
- u8 ch = console_utf_to_cp437(cp); uchar *pfont = fontdata->video_fontdata +
(u8)ch * fontdata->char_pixel_bytes;
ch * fontdata->char_pixel_bytes;if (x_frac + VID_TO_POS(vc_priv->x_charsize) > vc_priv->xsize_frac) return -EAGAIN;
diff --git a/drivers/video/console_truetype.c b/drivers/video/console_truetype.c index 14fb81e956..bc3ec1c31f 100644 --- a/drivers/video/console_truetype.c +++ b/drivers/video/console_truetype.c @@ -262,7 +262,7 @@ static int console_truetype_move_rows(struct udevice *dev, uint rowdst, }
static int console_truetype_putc_xy(struct udevice *dev, uint x, uint y,
char ch)
int cp)@@ -281,7 +281,7 @@ static int console_truetype_putc_xy(struct udevice *dev, uint x, uint y, int row, ret;
/* First get some basic metrics about this character */
- stbtt_GetCodepointHMetrics(font, ch, &advance, &lsb);
stbtt_GetCodepointHMetrics(font, cp, &advance, &lsb);
/*
- First out our current X position in fractional pixels. If we wrote
@@ -290,7 +290,7 @@ static int console_truetype_putc_xy(struct udevice *dev, uint x, uint y, xpos = frac(VID_TO_PIXEL((double)x)); if (vc_priv->last_ch) { xpos += met->scale * stbtt_GetCodepointKernAdvance(font,
vc_priv->last_ch, ch);
vc_priv->last_ch, cp);}
/*
@@ -320,7 +320,7 @@ static int console_truetype_putc_xy(struct udevice *dev, uint x, uint y, * return NULL; */ data = stbtt_GetCodepointBitmapSubpixel(font, met->scale, met->scale,
x_shift, 0, ch, &width, &height,
x_shift, 0, cp, &width, &height, &xoff, &yoff);diff --git a/drivers/video/vidconsole-uclass.c b/drivers/video/vidconsole-uclass.c index 22d55df71f..ce1668b97b 100644 --- a/drivers/video/vidconsole-uclass.c +++ b/drivers/video/vidconsole-uclass.c @@ -11,6 +11,7 @@
#include <common.h> #include <abuf.h> +#include <charset.h> #include <command.h> #include <console.h> #include <log.h> @@ -20,7 +21,7 @@ #include <video_font.h> /* Bitmap font for code page 437 */ #include <linux/ctype.h>
-int vidconsole_putc_xy(struct udevice *dev, uint x, uint y, char ch) +int vidconsole_putc_xy(struct udevice *dev, uint x, uint y, int ch) { struct vidconsole_ops *ops = vidconsole_get_ops(dev);
@@ -426,8 +427,8 @@ error: priv->escape = 0; }
-/* Put that actual character on the screen (using the CP437 code page). */ -static int vidconsole_output_glyph(struct udevice *dev, char ch) +/* Put that actual character on the screen (using the UTF-32 code points). */ +static int vidconsole_output_glyph(struct udevice *dev, int ch) { struct vidconsole_priv *priv = dev_get_uclass_priv(dev); int ret; @@ -455,7 +456,7 @@ static int vidconsole_output_glyph(struct udevice *dev, char ch) int vidconsole_put_char(struct udevice *dev, char ch)
This looks like a weird interface, to stuff in 8-bit bytes really, rather than the promised characters, and have some kind of internal state here. I am wondering if we should make this function static? There is only one real user (in board/atmel/common/video_display.c), that looks like it could use vidconsole_put_string(). Plus the tests, but we could fix them as well. Or we make this function also take a UTF-32?
Shouldn't hold back this series, but just stumbled upon it, and was curious what other people think.
{ struct vidconsole_priv *priv = dev_get_uclass_priv(dev);
- int ret;
int cp, ret;
if (priv->escape) { vidconsole_escape_char(dev, ch);
@@ -489,7 +490,14 @@ int vidconsole_put_char(struct udevice *dev, char ch) priv->last_ch = 0; break; default:
ret = vidconsole_output_glyph(dev, ch);
if (CONFIG_EFI_LOADER) {This would need to be: if (IS_ENABLED(CONFIG_EFI_LOADER)) {
Please, use CONFIG_IS_ENABLED(CHARSET). CONFIG_IS_ENABLED will correctly determine the avaialability of the symbol in SPL if we should ever have a video console there. Unicode is not only relevant for UEFI but also for correctly displaying non-ASCII file names. In general we should assume that code sequences used in the console are UTF-8 to support internationalization.
Otherwise disabling EFI_LOADER breaks compilation, since CONFIG_EFI_LOADER will not be defined at all:
drivers/video/vidconsole-uclass.c: In function ‘vidconsole_put_char’: drivers/video/vidconsole-uclass.c:493:7: error: ‘CONFIG_EFI_LOADER’ undeclared (first use in this function); did you mean ‘CONFIG_ENV_ADDR’?
But more general, is this really a compile time check? Without EFI applications, do we only expect 7-bit ASCII, or CP437 explicitly? It's only U-Boot generated output then, right?
A board on the Chinese market should be able to write a Chinese text to the console. This is not UEFI specific.
Best regards
Heinrich
cp = utf8_to_utf32_stream(ch, priv->utf8_buf);if (cp == 0)return 0;} else {cp = ch;}ret = vidconsole_output_glyph(dev, cp);diff --git a/drivers/video/vidconsole_internal.h b/drivers/video/vidconsole_internal.h index 0ec581b266..3d19410a52 100644 --- a/drivers/video/vidconsole_internal.h +++ b/drivers/video/vidconsole_internal.h @@ -6,6 +6,9 @@
- (C) Copyright 2023 Dzmitry Sankouski dsankouski@gmail.com
*/
+#include <charset.h> +#include <config.h>
- #define FLIPPED_DIRECTION 1 #define NORMAL_DIRECTION 0
@@ -142,3 +145,15 @@ int console_simple_get_font(struct udevice *dev, int seq, struct vidfont_info *i
- See details in video_console.h select_font function
**/ int console_simple_select_font(struct udevice *dev, const char *name, uint size);
+/**
- */
+static inline u8 console_utf_to_cp437(int codepoint) +{
- if (CONFIG_EFI_LOADER) {
Same IS_ENABLED() usage required here.
Cheers, Andre.
utf_to_cp(&codepoint, codepage_437);return (u8)codepoint;- }
- return (u8)codepoint;
+} diff --git a/include/video_console.h b/include/video_console.h index bde67fa9a5..20cf2d0101 100644 --- a/include/video_console.h +++ b/include/video_console.h @@ -43,6 +43,7 @@ enum {
- @col_saved: Saved X position, in fractional units (VID_TO_POS(x))
- @row_saved: Saved Y position in pixels (0=top)
- @escape_buf: Buffer to accumulate escape sequence
*/ struct vidconsole_priv { struct stdio_dev sdev;
- @utf8_buf: Buffer to accumulate UTF-8 byte sequence
@@ -66,6 +67,7 @@ struct vidconsole_priv { int row_saved; int col_saved; char escape_buf[32];
char utf8_buf[8]; };
/**
@@ -124,12 +126,12 @@ struct vidconsole_ops { * @x_frac: Fractional pixel X position (0=left-most pixel) which * is the X position multipled by VID_FRAC_DIV. * @y: Pixel Y position (0=top-most pixel)
* @ch: Character to write
* @cp: EF UTF-32 code point to write*/
- @return number of fractional pixels that the cursor should move,
- if all is OK, -EAGAIN if we ran out of space on this line, other -ve
- on error
- int (*putc_xy)(struct udevice *dev, uint x_frac, uint y, char ch);
int (*putc_xy)(struct udevice *dev, uint x_frac, uint y, int cp);
/**
- move_rows() - Move text rows from one place to another
@@ -403,12 +405,12 @@ void vidconsole_pop_colour(struct udevice *dev, struct vidconsole_colour *old);
- @x_frac: Fractional pixel X position (0=left-most pixel) which
is the X position multipled by VID_FRAC_DIV.- @y: Pixel Y position (0=top-most pixel)
- @ch: Character to write
*/
- @cp: UTF-32 code point to write
- Return: number of fractional pixels that the cursor should move,
- if all is OK, -EAGAIN if we ran out of space on this line, other -ve
- on error
-int vidconsole_putc_xy(struct udevice *dev, uint x, uint y, char ch); +int vidconsole_putc_xy(struct udevice *dev, uint x, uint y, int cp);
/**
- vidconsole_move_rows() - Move text rows from one place to another
From: Janne Grunau j@jannau.net
Add mappings for code points 1 - 31 as those code points in code page 437 are graphics. Thios fixes rendering issues of various EFI boot loaders (grub2, sd-boot, ...) using EFI_SIMPLE_TEXT_OUTPUT_PROTOCOL.
Signed-off-by: Janne Grunau j@jannau.net --- include/charset.h | 2 +- include/cp1250.h | 12 ++++++++++-- include/cp437.h | 12 ++++++++++-- lib/charset.c | 9 ++++++--- lib/efi_loader/efi_unicode_collation.c | 2 +- 5 files changed, 28 insertions(+), 9 deletions(-)
diff --git a/include/charset.h b/include/charset.h index 714382e1c1..c51c29235f 100644 --- a/include/charset.h +++ b/include/charset.h @@ -16,7 +16,7 @@ /* * codepage_437 - Unicode to codepage 437 translation table */ -extern const u16 codepage_437[128]; +extern const u16 codepage_437[160];
/** * console_read_unicode() - read Unicode code point from console diff --git a/include/cp1250.h b/include/cp1250.h index adacf8a958..b762c78d9f 100644 --- a/include/cp1250.h +++ b/include/cp1250.h @@ -1,10 +1,18 @@ /* SPDX-License-Identifier: GPL-2.0+ */
/* - * Constant CP1250 contains the Unicode code points for characters 0x80 - 0xff - * of the code page 1250. + * Constant CP1250 contains the Unicode code points for characters 0x00 - 0x1f + * and 0x80 - 0xff of the code page 1250. */ #define CP1250 { \ + 0x0000, 0x0000, 0x0000, 0x0000, \ + 0x0000, 0x0000, 0x0000, 0x0000, \ + 0x0000, 0x0000, 0x0000, 0x0000, \ + 0x0000, 0x0000, 0x0000, 0x0000, \ + 0x0000, 0x0000, 0x0000, 0x0000, \ + 0x0000, 0x0000, 0x0000, 0x0000, \ + 0x0000, 0x0000, 0x0000, 0x0000, \ + 0x0000, 0x0000, 0x0000, 0x0000, \ 0x20ac, 0x0000, 0x201a, 0x0000, \ 0x201e, 0x2026, 0x2020, 0x2021, \ 0x0000, 0x2030, 0x0160, 0x2039, \ diff --git a/include/cp437.h b/include/cp437.h index 0b2b97132e..5093130f5e 100644 --- a/include/cp437.h +++ b/include/cp437.h @@ -1,10 +1,18 @@ /* SPDX-License-Identifier: GPL-2.0+ */
/* - * Constant CP437 contains the Unicode code points for characters 0x80 - 0xff - * of the code page 437. + * Constant CP437 contains the Unicode code points for characters 0x00 - 0x1f + * and 0x80 - 0xff of the code page 437. */ #define CP437 { \ + 0x0000, 0x263a, 0x263b, 0x2665, \ + 0x2666, 0x2663, 0x2660, 0x2022, \ + 0x25d8, 0x25cb, 0x25d9, 0x2642, \ + 0x2640, 0x266a, 0x266b, 0x263c, \ + 0x25ba, 0x25c4, 0x2195, 0x203c, \ + 0x00b6, 0x00a7, 0x25ac, 0x21a8, \ + 0x2191, 0x2193, 0x2192, 0x2190, \ + 0x221f, 0x2194, 0x25b2, 0x25bc, \ 0x00c7, 0x00fc, 0x00e9, 0x00e2, \ 0x00e4, 0x00e0, 0x00e5, 0x00e7, \ 0x00ea, 0x00eb, 0x00e8, 0x00ef, \ diff --git a/lib/charset.c b/lib/charset.c index 5e4c4f948a..1f8480150a 100644 --- a/lib/charset.c +++ b/lib/charset.c @@ -16,7 +16,7 @@ /** * codepage_437 - Unicode to codepage 437 translation table */ -const u16 codepage_437[128] = CP437; +const u16 codepage_437[160] = CP437;
static struct capitalization_table capitalization_table[] = #ifdef CONFIG_EFI_UNICODE_CAPITALIZATION @@ -517,9 +517,12 @@ int utf_to_cp(s32 *c, const u16 *codepage) int j;
/* Look up codepage translation */ - for (j = 0; j < 0x80; ++j) { + for (j = 0; j < 0xA0; ++j) { if (*c == codepage[j]) { - *c = j + 0x80; + if (j < 0x20) + *c = j; + else + *c = j + 0x60; return 0; } } diff --git a/lib/efi_loader/efi_unicode_collation.c b/lib/efi_loader/efi_unicode_collation.c index c4c7572063..4b2c52918a 100644 --- a/lib/efi_loader/efi_unicode_collation.c +++ b/lib/efi_loader/efi_unicode_collation.c @@ -257,7 +257,7 @@ static void EFIAPI efi_fat_to_str(struct efi_unicode_collation_protocol *this, for (i = 0; i < fat_size; ++i) { c = (unsigned char)fat[i]; if (c > 0x80) - c = codepage[c - 0x80]; + c = codepage[c - 0x60]; string[i] = c; if (!c) break;
On 1/17/24 23:24, Janne Grunau via B4 Relay wrote:
From: Janne Grunau j@jannau.net
Add mappings for code points 1 - 31 as those code points in code page 437 are graphics. Thios fixes rendering issues of various EFI boot loaders (grub2,
%s/Thios/This/
sd-boot, ...) using EFI_SIMPLE_TEXT_OUTPUT_PROTOCOL.
Could you, please, explain the reason for the change a bit more in depth.
The applications above produce UTF-16. A Truetype font should not have symbols for positions 0x00-0x1f. Codepage 437 is only relevant for FAT file names that might use this character set.
Why do you want to translate well define Unicode code points to characters 0x00-0x1f?
Best regards
Heinrich
Signed-off-by: Janne Grunau j@jannau.net
include/charset.h | 2 +- include/cp1250.h | 12 ++++++++++-- include/cp437.h | 12 ++++++++++-- lib/charset.c | 9 ++++++--- lib/efi_loader/efi_unicode_collation.c | 2 +- 5 files changed, 28 insertions(+), 9 deletions(-)
diff --git a/include/charset.h b/include/charset.h index 714382e1c1..c51c29235f 100644 --- a/include/charset.h +++ b/include/charset.h @@ -16,7 +16,7 @@ /*
- codepage_437 - Unicode to codepage 437 translation table
*/ -extern const u16 codepage_437[128]; +extern const u16 codepage_437[160];
/**
- console_read_unicode() - read Unicode code point from console
diff --git a/include/cp1250.h b/include/cp1250.h index adacf8a958..b762c78d9f 100644 --- a/include/cp1250.h +++ b/include/cp1250.h @@ -1,10 +1,18 @@ /* SPDX-License-Identifier: GPL-2.0+ */
/*
- Constant CP1250 contains the Unicode code points for characters 0x80 - 0xff
- of the code page 1250.
- Constant CP1250 contains the Unicode code points for characters 0x00 - 0x1f
*/ #define CP1250 { \
- and 0x80 - 0xff of the code page 1250.
- 0x0000, 0x0000, 0x0000, 0x0000, \
- 0x0000, 0x0000, 0x0000, 0x0000, \
- 0x0000, 0x0000, 0x0000, 0x0000, \
- 0x0000, 0x0000, 0x0000, 0x0000, \
- 0x0000, 0x0000, 0x0000, 0x0000, \
- 0x0000, 0x0000, 0x0000, 0x0000, \
- 0x0000, 0x0000, 0x0000, 0x0000, \
- 0x0000, 0x0000, 0x0000, 0x0000, \ 0x20ac, 0x0000, 0x201a, 0x0000, \ 0x201e, 0x2026, 0x2020, 0x2021, \ 0x0000, 0x2030, 0x0160, 0x2039, \
diff --git a/include/cp437.h b/include/cp437.h index 0b2b97132e..5093130f5e 100644 --- a/include/cp437.h +++ b/include/cp437.h @@ -1,10 +1,18 @@ /* SPDX-License-Identifier: GPL-2.0+ */
/*
- Constant CP437 contains the Unicode code points for characters 0x80 - 0xff
- of the code page 437.
- Constant CP437 contains the Unicode code points for characters 0x00 - 0x1f
*/ #define CP437 { \
- and 0x80 - 0xff of the code page 437.
- 0x0000, 0x263a, 0x263b, 0x2665, \
- 0x2666, 0x2663, 0x2660, 0x2022, \
- 0x25d8, 0x25cb, 0x25d9, 0x2642, \
- 0x2640, 0x266a, 0x266b, 0x263c, \
- 0x25ba, 0x25c4, 0x2195, 0x203c, \
- 0x00b6, 0x00a7, 0x25ac, 0x21a8, \
- 0x2191, 0x2193, 0x2192, 0x2190, \
- 0x221f, 0x2194, 0x25b2, 0x25bc, \ 0x00c7, 0x00fc, 0x00e9, 0x00e2, \ 0x00e4, 0x00e0, 0x00e5, 0x00e7, \ 0x00ea, 0x00eb, 0x00e8, 0x00ef, \
diff --git a/lib/charset.c b/lib/charset.c index 5e4c4f948a..1f8480150a 100644 --- a/lib/charset.c +++ b/lib/charset.c @@ -16,7 +16,7 @@ /**
- codepage_437 - Unicode to codepage 437 translation table
*/ -const u16 codepage_437[128] = CP437; +const u16 codepage_437[160] = CP437;
static struct capitalization_table capitalization_table[] = #ifdef CONFIG_EFI_UNICODE_CAPITALIZATION @@ -517,9 +517,12 @@ int utf_to_cp(s32 *c, const u16 *codepage) int j;
/* Look up codepage translation */
for (j = 0; j < 0x80; ++j) {
for (j = 0; j < 0xA0; ++j) { if (*c == codepage[j]) {
*c = j + 0x80;
if (j < 0x20)*c = j;else*c = j + 0x60; return 0; }diff --git a/lib/efi_loader/efi_unicode_collation.c b/lib/efi_loader/efi_unicode_collation.c index c4c7572063..4b2c52918a 100644 --- a/lib/efi_loader/efi_unicode_collation.c +++ b/lib/efi_loader/efi_unicode_collation.c @@ -257,7 +257,7 @@ static void EFIAPI efi_fat_to_str(struct efi_unicode_collation_protocol *this, for (i = 0; i < fat_size; ++i) { c = (unsigned char)fat[i]; if (c > 0x80)
c = codepage[c - 0x80];
c = codepage[c - 0x60];
From: Andre Przywara andre.przywara@arm.com
UEFI relies entirely on unicode output, which actual fonts displayed on the screen might not be ready for.
Add a test displaying some international characters, to reveal missing glyphs, especially in our builtin fonts. This would be needed to be manually checked on the screen for correctness.
Signed-off-by: Andre Przywara andre.przywara@arm.com Suggested-by: Heinrich Schuchardt xypron.glpk@gmx.de Signed-off-by: Janne Grunau j@jannau.net --- lib/efi_selftest/efi_selftest_textoutput.c | 8 ++++++++ 1 file changed, 8 insertions(+)
diff --git a/lib/efi_selftest/efi_selftest_textoutput.c b/lib/efi_selftest/efi_selftest_textoutput.c index cc44b38bc2..2aa81b0a80 100644 --- a/lib/efi_selftest/efi_selftest_textoutput.c +++ b/lib/efi_selftest/efi_selftest_textoutput.c @@ -30,6 +30,9 @@ static int execute(void) 0xD804, 0xDC05, 0xD804, 0xDC22, 0}; + const u16 text[] = +u"\u00d6sterreich Edelwei\u00df Sm\u00f8rrebr\u00f8d Sm\u00f6rg" +u"\u00e5s Ni\u00f1o Ren\u00e9 >\u1f19\u03bb\u03bb\u03ac\u03c2<\n";
/* SetAttribute */ efi_st_printf("\nColor palette\n"); @@ -118,6 +121,11 @@ static int execute(void) efi_st_printf("Unicode not handled properly\n"); return EFI_ST_FAILURE; } + ret = con_out->output_string(con_out, text); + if (ret != EFI_ST_SUCCESS) { + efi_st_error("OutputString failed for international chars\n"); + return EFI_ST_FAILURE; + } efi_st_printf("\n");
return EFI_ST_SUCCESS;
On 1/17/24 23:24, Janne Grunau via B4 Relay wrote:
From: Andre Przywara andre.przywara@arm.com
UEFI relies entirely on unicode output, which actual fonts displayed on the screen might not be ready for.
Add a test displaying some international characters, to reveal missing glyphs, especially in our builtin fonts.
How would I detect missing Bengali and Gujarati glyphs with this test?
What might help would be an output of the Unicode entries in the font in parallel with the glyph.
U+0040, @ U+0041, A U+0A85. અ U+0A86. આ U+0A87. ઇ U+0A88. ઈ U+0A89. ઉ
This would require some new command but not an EFI test.
Best regards
Heinrich
This would be needed to be manually checked on the screen for correctness.
Signed-off-by: Andre Przywara andre.przywara@arm.com Suggested-by: Heinrich Schuchardt xypron.glpk@gmx.de Signed-off-by: Janne Grunau j@jannau.net
lib/efi_selftest/efi_selftest_textoutput.c | 8 ++++++++ 1 file changed, 8 insertions(+)
diff --git a/lib/efi_selftest/efi_selftest_textoutput.c b/lib/efi_selftest/efi_selftest_textoutput.c index cc44b38bc2..2aa81b0a80 100644 --- a/lib/efi_selftest/efi_selftest_textoutput.c +++ b/lib/efi_selftest/efi_selftest_textoutput.c @@ -30,6 +30,9 @@ static int execute(void) 0xD804, 0xDC05, 0xD804, 0xDC22, 0};
- const u16 text[] =
+u"\u00d6sterreich Edelwei\u00df Sm\u00f8rrebr\u00f8d Sm\u00f6rg" +u"\u00e5s Ni\u00f1o Ren\u00e9 >\u1f19\u03bb\u03bb\u03ac\u03c2<\n";
/* SetAttribute */ efi_st_printf("\nColor palette\n"); @@ -118,6 +121,11 @@ static int execute(void) efi_st_printf("Unicode not handled properly\n"); return EFI_ST_FAILURE; }
ret = con_out->output_string(con_out, text);
if (ret != EFI_ST_SUCCESS) {
efi_st_error("OutputString failed for international chars\n");return EFI_ST_FAILURE;} efi_st_printf("\n");
return EFI_ST_SUCCESS;
Hej,
On Thu, Jan 18, 2024, at 19:48, Heinrich Schuchardt wrote:
On 1/17/24 23:24, Janne Grunau via B4 Relay wrote:
From: Andre Przywara andre.przywara@arm.com
UEFI relies entirely on unicode output, which actual fonts displayed on the screen might not be ready for.
Add a test displaying some international characters, to reveal missing glyphs, especially in our builtin fonts.
How would I detect missing Bengali and Gujarati glyphs with this test?
Only by knowing what to expect or to check the test code.
What might help would be an output of the Unicode entries in the font in parallel with the glyph.
I'm not sure how useful it would be to extend the amount of text to register during the test. Even with the 3 second wait added in the the box drawing test time is short to check the results. Running the tests optionally with pagination might be an option.
U+0040, @ U+0041, A U+0A85. અ U+0A86. આ U+0A87. ઇ U+0A88. ઈ U+0A89. ઉThis would require some new command but not an EFI test.
Wouldn't we need both? Console tests to check if the intended glyphs are shown would be useful but the EFI selftest should still tests that it can render UTF-8 correctly (partial support on cp437 bitmap font consoles).
Best regards Janne
From: Andre Przywara andre.przywara@arm.com
UEFI applications rely on Unicode output capability, and might use that for drawing pseudo-graphical interfaces using Unicode defined box drawing characters.
Add a simple test to display the most basic box characters, which would need to be checked manually on the screen for correctness. To facilitate this, add a three second delay after the output at this point.
Signed-off-by: Andre Przywara andre.przywara@arm.com Suggested-by: Heinrich Schuchardt xypron.glpk@gmx.de Signed-off-by: Janne Grunau j@jannau.net --- lib/efi_selftest/efi_selftest_textoutput.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+)
diff --git a/lib/efi_selftest/efi_selftest_textoutput.c b/lib/efi_selftest/efi_selftest_textoutput.c index 2aa81b0a80..cc11a22eee 100644 --- a/lib/efi_selftest/efi_selftest_textoutput.c +++ b/lib/efi_selftest/efi_selftest_textoutput.c @@ -33,6 +33,19 @@ static int execute(void) const u16 text[] = u"\u00d6sterreich Edelwei\u00df Sm\u00f8rrebr\u00f8d Sm\u00f6rg" u"\u00e5s Ni\u00f1o Ren\u00e9 >\u1f19\u03bb\u03bb\u03ac\u03c2<\n"; + const u16 boxes[] = +u"This should render as four boxes with text\n" +u"\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500" +u"\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500" +u"\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502" +u" left top \u2502 right top \u2502\n\u251c\u2500" +u"\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500" +u"\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500" +u"\u2500\u2500\u2500\u2500\u2500\u2500\u2524\n\u2502 " +u"left bottom \u2502 right bottom \u2502\n\u2514\u2500\u2500\u2500" +u"\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534" +u"\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500" +u"\u2500\u2500\u2500\u2500\u2518\n";
/* SetAttribute */ efi_st_printf("\nColor palette\n"); @@ -126,6 +139,13 @@ u"\u00e5s Ni\u00f1o Ren\u00e9 >\u1f19\u03bb\u03bb\u03ac\u03c2<\n"; efi_st_error("OutputString failed for international chars\n"); return EFI_ST_FAILURE; } + ret = con_out->output_string(con_out, boxes); + if (ret != EFI_ST_SUCCESS) { + efi_st_error("OutputString failed for box drawing chars\n"); + return EFI_ST_FAILURE; + } + con_out->output_string(con_out, u"waiting for admiration...\n"); + EFI_CALL(systab.boottime->stall(3000000)); efi_st_printf("\n");
return EFI_ST_SUCCESS;
From: Janne Grunau j@jannau.net
Draw symbols from code page 437 code points 0x01 - 01f used by UEFI applications to draw user interfaces using EFI_SIMPLE_TEXT_OUTPUT_PROTOCOL.
Add a simple test to displaying the Konami code using symbols used by grub2. The output has to be checked manually on the screen for correctness.
Signed-off-by: Janne Grunau j@jannau.net --- lib/efi_selftest/efi_selftest_textoutput.c | 7 +++++++ 1 file changed, 7 insertions(+)
diff --git a/lib/efi_selftest/efi_selftest_textoutput.c b/lib/efi_selftest/efi_selftest_textoutput.c index cc11a22eee..9e5d944fa0 100644 --- a/lib/efi_selftest/efi_selftest_textoutput.c +++ b/lib/efi_selftest/efi_selftest_textoutput.c @@ -46,6 +46,8 @@ u"left bottom \u2502 right bottom \u2502\n\u2514\u2500\u2500\u2500" u"\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534" u"\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500" u"\u2500\u2500\u2500\u2500\u2518\n"; + const u16 konami[] = +u"Konami code: \u25b2 \u25b2 \u25bc \u25bc \u25c4 \u25ba \u25c4 \u25ba B A\n";
/* SetAttribute */ efi_st_printf("\nColor palette\n"); @@ -144,6 +146,11 @@ u"\u2500\u2500\u2500\u2500\u2518\n"; efi_st_error("OutputString failed for box drawing chars\n"); return EFI_ST_FAILURE; } + ret = con_out->output_string(con_out, konami); + if (ret != EFI_ST_SUCCESS) { + efi_st_error("OutputString failed for symbol chars\n"); + return EFI_ST_FAILURE; + } con_out->output_string(con_out, u"waiting for admiration...\n"); EFI_CALL(systab.boottime->stall(3000000)); efi_st_printf("\n");
On Wed, 17 Jan 2024 23:24:26 +0100 Janne Grunau via B4 Relay devnull+j.jannau.net@kernel.org wrote:
Hi Janne,
Andre submitted 2 years ago DM_VIDEO improvements for UEFI applications using UEFI's EFI_SIMPLE_TEXT_OUTPUT_PROTOCOL but did not follow with suggested changes. This series takes care of the UTF-8 support which required to draw symbol and box drawing characters used by UEFI applications like grub2 and sd-boot correctly.
Compared to Andre's version this version has the following changes:
- use and extend existing conversion functions from lib/charset.c
- convert first to UTF-32 to support the truetype console as well
- conversion is conditional on CONFIG_EFI_LOADER
- use escape sequences in tests as proposed by Heinrich
many thanks for picking this up, much appreciated! In general this looks good, I will test this later this week, and have a closer look, once I am on a device with video output.
Cheers, Andre
Link: https://lore.kernel.org/u-boot/20220110005638.21599-1-andre.przywara@arm.com... Signed-off-by: Janne Grunau j@jannau.net
Andre Przywara (2): efi_selftest: Add international characters test efi_selftest: Add box drawing character selftest
Janne Grunau (4): lib: charset: Fix utf8_to_utf32_stream() return value doc string video: console: Parse UTF-8 character sequences lib/charset: Map cp437 low chars (0x01 - 0x1f) from unicode efi_selftest: Add symbol character selftest
drivers/video/console_normal.c | 6 +++-- drivers/video/console_rotate.c | 16 +++++++++----- drivers/video/console_truetype.c | 8 +++---- drivers/video/vidconsole-uclass.c | 18 ++++++++++----- drivers/video/vidconsole_internal.h | 15 +++++++++++++ include/charset.h | 4 ++-- include/cp1250.h | 12 ++++++++-- include/cp437.h | 12 ++++++++-- include/video_console.h | 10 +++++---- lib/charset.c | 9 +++++--- lib/efi_loader/efi_unicode_collation.c | 2 +- lib/efi_selftest/efi_selftest_textoutput.c | 35 ++++++++++++++++++++++++++++++ 12 files changed, 116 insertions(+), 31 deletions(-)
base-commit: 866ca972d6c3cabeaf6dbac431e8e08bb30b3c8e change-id: 20240117-vidconsole-utf8-uefi-fa23b4ac65d6
Best regards,
On 1/17/24 23:24, Janne Grunau via B4 Relay wrote:
Andre submitted 2 years ago DM_VIDEO improvements for UEFI applications using UEFI's EFI_SIMPLE_TEXT_OUTPUT_PROTOCOL but did not follow with suggested changes. This series takes care of the UTF-8 support which required to draw symbol and box drawing characters used by UEFI applications like grub2 and sd-boot correctly.
Compared to Andre's version this version has the following changes:
- use and extend existing conversion functions from lib/charset.c
- convert first to UTF-32 to support the truetype console as well
- conversion is conditional on CONFIG_EFI_LOADER
- use escape sequences in tests as proposed by Heinrich
Hello Janne,
to correctly render GRUB I think you need the following:
* Use truetype * Add a mono-spaced Truetype font which has all the needed characters. * A bunch of code fixed.
This is the font I once suggested.
[PATCH v2 1/1] video: add DejaVu Mono font https://lore.kernel.org/u-boot/20210301181534.7618-1-xypron.glpk@gmx.de/
Which font have you been using for testing?
At the time these additional patches were needed.
[PATCH 0/6] efi_loader: Unicode output in UEFI applications https://lore.kernel.org/u-boot/20210227130840.166193-1-xypron.glpk@gmx.de/#r
Best regards
Heinrich
Link: https://lore.kernel.org/u-boot/20220110005638.21599-1-andre.przywara@arm.com... Signed-off-by: Janne Grunau j@jannau.net
Andre Przywara (2): efi_selftest: Add international characters test efi_selftest: Add box drawing character selftest
Janne Grunau (4): lib: charset: Fix utf8_to_utf32_stream() return value doc string video: console: Parse UTF-8 character sequences lib/charset: Map cp437 low chars (0x01 - 0x1f) from unicode efi_selftest: Add symbol character selftest
drivers/video/console_normal.c | 6 +++-- drivers/video/console_rotate.c | 16 +++++++++----- drivers/video/console_truetype.c | 8 +++---- drivers/video/vidconsole-uclass.c | 18 ++++++++++----- drivers/video/vidconsole_internal.h | 15 +++++++++++++ include/charset.h | 4 ++-- include/cp1250.h | 12 ++++++++-- include/cp437.h | 12 ++++++++-- include/video_console.h | 10 +++++---- lib/charset.c | 9 +++++--- lib/efi_loader/efi_unicode_collation.c | 2 +- lib/efi_selftest/efi_selftest_textoutput.c | 35 ++++++++++++++++++++++++++++++ 12 files changed, 116 insertions(+), 31 deletions(-)
base-commit: 866ca972d6c3cabeaf6dbac431e8e08bb30b3c8e change-id: 20240117-vidconsole-utf8-uefi-fa23b4ac65d6
Best regards,
Hej Heinrich,
On Thu, Jan 18, 2024, at 17:47, Heinrich Schuchardt wrote:
On 1/17/24 23:24, Janne Grunau via B4 Relay wrote:
Andre submitted 2 years ago DM_VIDEO improvements for UEFI applications using UEFI's EFI_SIMPLE_TEXT_OUTPUT_PROTOCOL but did not follow with suggested changes. This series takes care of the UTF-8 support which required to draw symbol and box drawing characters used by UEFI applications like grub2 and sd-boot correctly.
Compared to Andre's version this version has the following changes:
- use and extend existing conversion functions from lib/charset.c
- convert first to UTF-32 to support the truetype console as well
- conversion is conditional on CONFIG_EFI_LOADER
- use escape sequences in tests as proposed by Heinrich
Hello Janne,
to correctly render GRUB I think you need the following:
- Use truetype
a truetype font is not necessary, the arrow glyphs used by grub are in codepage 437 (code points 0x10 - 0x1f, see https://en.wikipedia.org/wiki/Code_page_437#Character_set ). grub2 and sd-boot render correctly with this patchset and bitmap fonts.
- Add a mono-spaced Truetype font which has all the needed characters.
- A bunch of code fixed.
This is the font I once suggested.
[PATCH v2 1/1] video: add DejaVu Mono font https://lore.kernel.org/u-boot/20210301181534.7618-1-xypron.glpk@gmx.de/
Which font have you been using for testing?
The 8x16 and 16x32 bitmap fonts and the included Anker/Coder Narrow truetype font. All render the grub menu correctly but the bitmap fonts are missing some of the glyphs in the international character efi_selftest by Andre and you.
At the time these additional patches were needed.
[PATCH 0/6] efi_loader: Unicode output in UEFI applications https://lore.kernel.org/u-boot/20210227130840.166193-1-xypron.glpk@gmx.de/#r
That is functionally identical to this patchset except for the Unicode mapping for code page 437 code points 0x01 to 0x1f (Patch 3/6 "lib/charset: Map cp437 low chars (0x01 - 0x1f) from unicode"). I wasn't aware of this and stopped looking after Andre's patches. It looks like patches 5 and 6 were never merged. Was there a reason for that? best regards Janne
Link: https://lore.kernel.org/u-boot/20220110005638.21599-1-andre.przywara@arm.com... Signed-off-by: Janne Grunau j@jannau.net
Andre Przywara (2): efi_selftest: Add international characters test efi_selftest: Add box drawing character selftest
Janne Grunau (4): lib: charset: Fix utf8_to_utf32_stream() return value doc string video: console: Parse UTF-8 character sequences lib/charset: Map cp437 low chars (0x01 - 0x1f) from unicode efi_selftest: Add symbol character selftest
drivers/video/console_normal.c | 6 +++-- drivers/video/console_rotate.c | 16 +++++++++----- drivers/video/console_truetype.c | 8 +++---- drivers/video/vidconsole-uclass.c | 18 ++++++++++----- drivers/video/vidconsole_internal.h | 15 +++++++++++++ include/charset.h | 4 ++-- include/cp1250.h | 12 ++++++++-- include/cp437.h | 12 ++++++++-- include/video_console.h | 10 +++++---- lib/charset.c | 9 +++++--- lib/efi_loader/efi_unicode_collation.c | 2 +- lib/efi_selftest/efi_selftest_textoutput.c | 35 ++++++++++++++++++++++++++++++ 12 files changed, 116 insertions(+), 31 deletions(-)
base-commit: 866ca972d6c3cabeaf6dbac431e8e08bb30b3c8e change-id: 20240117-vidconsole-utf8-uefi-fa23b4ac65d6
Best regards,
participants (4)
-
Andre Przywara -
Heinrich Schuchardt -
Janne Grunau -
Janne Grunau via B4 Relay