View Issue Details
ID | Project | Category | View Status | Date Submitted | Last Update |
---|---|---|---|---|---|
0000786 | Far Manager | Viewer | public | 2009-03-13 14:35 | 2011-03-21 23:56 |
Reporter | Denis | Assigned To | vskirdin | ||
Priority | normal | Severity | major | Reproducibility | always |
Status | closed | Resolution | fixed | ||
Fixed in Version | 3.0 | ||||
Summary | 0000786: Неверное отображение найденного текста в файлах UTF-8 | ||||
Description | Поиск в UTF-8 файлах с русским текстом не работает. | ||||
Additional Information | Far2 823 | ||||
Tags | No tags attached. | ||||
Build | 1903 | ||||
has duplicate | 0000977 | closed | Неправильный поиск русского текста в кодировке UTF-8 | |
has duplicate | 0001173 | closed | alexy | Неправильное позицирование курсора при поиске в тексте |
has duplicate | 0001707 | closed | alexy | Разный результат поиска при разных начальных позициях |
has duplicate | 0001562 | closed | alexy | Viewer does not find string which she must find |
has duplicate | 0001563 | closed | Viewer highlights found string incorrectly | |
has duplicate | 0001216 | closed | Search does not work for UNICODE files | |
has duplicate | 0001749 | closed | Некорректная работа поиска в юникодных кодовых страницах | |
has duplicate | 0001761 | closed | Viewer fails to search in .fb2 file, while Editor does. По F3 в fb2 файлах поиск не работает, а по F4 работает | |
has duplicate | 0001900 | closed | 2useven10 | Search in viewing of UTF-8 file does not find string file Does contain, ANSI and OEM works |
related to | 0001047 | closed | Неправильно находится смещние текста при поиске в просмотре (UTF-8, если есть русский текст) | |
related to | 0001046 | closed | Неправильно находится смещние текста при поиске в просмотре (UTF-8, если есть русский текст) |
|
Не работает, если выбраны все кодировки. Когда выбрана UTF-8 — находит. Far2 823, 825. |
|
Баг не связан с окном поиска по Alt-F7, а только с Viewer'ом. |
|
Wave, заводи новый баг и давай последовательность воспроизведения. У меня по UTF-8 ищет, как при прямом выборе, так и при посике по всем кодировкам. |
2009-04-16 09:03
|
|
|
870 не работает 1. F3 на FarRus.hlf 2. F7, ввод "plugins", Enter Находит после 3-4 нажатий пробела и искажает верхние строки (см. приложение) |
|
Нужно поправить description на "Неверное отображение найденного текста в файлах UTF-8" |
|
либо находит лишний раз строку, либо не находит строку вообще. см мой пример в баге 977 |
|
Отлично проявляется на po-файлах. build 1184 x86. |
|
каждую неделю постят и постят сабж, может приделать сообщение на F7 - не работает! например, как в аттаче. |
|
!work.diff (2,280 bytes)
Index: viewer.cpp =================================================================== --- viewer.cpp (revision 5207) +++ viewer.cpp (working copy) @@ -1356,20 +1356,31 @@ } case KEY_F7: { - Search(0,0); + if (VM.CodePage==CP_UTF8) + Message(MSG_WARNING,1,MSG(MViewSearchTitle),MSG(MViewSearchNotSupportUTF8),MSG(MOk)); + else + Search(0,0); return TRUE; } case KEY_SHIFTF7: case KEY_SPACE: { - Search(1,0); + if (VM.CodePage==CP_UTF8) + Message(MSG_WARNING,1,MSG(MViewSearchTitle),MSG(MViewSearchNotSupportUTF8),MSG(MOk)); + else + Search(1,0); return TRUE; } case KEY_ALTF7: { - SearchFlags.Set(REVERSE_SEARCH); - Search(1,0); - SearchFlags.Clear(REVERSE_SEARCH); + if (VM.CodePage==CP_UTF8) + Message(MSG_WARNING,1,MSG(MViewSearchTitle),MSG(MViewSearchNotSupportUTF8),MSG(MOk)); + else + { + SearchFlags.Set(REVERSE_SEARCH); + Search(1,0); + SearchFlags.Clear(REVERSE_SEARCH); + } return TRUE; } case KEY_F8: Index: farlang.templ.m4 =================================================================== --- farlang.templ.m4 (revision 5207) +++ farlang.templ.m4 (working copy) @@ -13992,6 +13992,15 @@ "Kontynuować wyszukiwanie od końca dokumentu?" "Continuar búsqueda desde el final del documento" +MViewSearchNotSupportUTF8 +"Поиск в программе просмотра в кодовой странице UTF-8 не реализован" +"Поиск в программе просмотра в кодовой странице UTF-8 не реализован" +"Поиск в программе просмотра в кодовой странице UTF-8 не реализован" +"Поиск в программе просмотра в кодовой странице UTF-8 не реализован" +"Поиск в программе просмотра в кодовой странице UTF-8 не реализован" +"Поиск в программе просмотра в кодовой странице UTF-8 не реализован" +"Поиск в программе просмотра в кодовой странице UTF-8 не реализован" + MPrintTitle l: "Печать" |
|
Кому и сколько заплатить, чтобы поиск в UTF-8 при просмотре заработал? (это не шутка) |
|
viewer.diff (9,730 bytes)
Index: cache.hpp =================================================================== --- cache.hpp (revision 5769) +++ cache.hpp (working copy) @@ -39,6 +39,7 @@ ~CachedRead(); bool Read(LPVOID Data, DWORD DataSize, LPDWORD BytesRead); bool FillBuffer(); + bool Unread(DWORD BytesUnread); void Clear(); private: Index: viewer.hpp =================================================================== --- viewer.hpp (revision 5769) +++ viewer.hpp (working copy) @@ -163,6 +163,9 @@ bool m_bQuickView; UINT DefCodePage; + + int vgetc_ready; + private: virtual void DisplayObject(); @@ -187,7 +190,7 @@ void Search(int Next,int FirstChar); void ConvertToHex(char *SearchStr,int &SearchLength); int HexToNum(int Hex); - int vread(wchar_t *Buf,int Count, bool Raw=false); + int vread(wchar_t *Buf,int Count, bool Raw=false, wchar_t *Buf2=NULL); int vseek(__int64 Offset,int Whence); __int64 vtell(); bool vgetc(WCHAR& C); Index: cache.cpp =================================================================== --- cache.cpp (revision 5769) +++ cache.cpp (working copy) @@ -109,6 +109,18 @@ return Result; } +bool CachedRead::Unread(DWORD BytesUnread) +{ + if (BytesUnread + BytesLeft <= ReadSize) + { + BytesLeft += BytesUnread; + __int64 off = BytesUnread; + file.SetPointer(-off, &LastPtr, FILE_CURRENT); + return true; + } + return false; +} + bool CachedRead::FillBuffer() { bool Result=false; Index: viewer.cpp =================================================================== --- viewer.cpp (revision 5769) +++ viewer.cpp (working copy) @@ -122,6 +122,7 @@ SelectPosOffSet=0; bVE_READ_Sent = false; Signature = false; + vgetc_ready = -1; } @@ -1929,9 +1930,12 @@ if (!ViewFile.Opened()) return; - wchar_t Buf[MAX_VIEWLINE]; + wchar_t Buf[MAX_VIEWLINE], Buf2[ARRAYSIZE(Buf)], *tBuf = Buf, *tBuf2 = NULL; int BufSize,StrPos,Skipped,I,J; + if (!VM.Hex && VM.CodePage == CP_UTF8) + tBuf = tBuf2 = Buf2; + if (FilePos > (__int64)sizeof(Buf)/sizeof(wchar_t)) BufSize=sizeof(Buf)/sizeof(wchar_t); else @@ -1955,7 +1959,7 @@ } vseek(FilePos-(__int64)BufSize,SEEK_SET); - BufSize = vread(Buf,BufSize); + BufSize = vread(Buf, BufSize, false, tBuf2); Skipped=0; if (BufSize>0 && Buf[BufSize-1]==(unsigned int)CRSym) @@ -1979,7 +1983,7 @@ { if (!VM.Wrap) { - FilePos -= GetStrBytesNum(Buf + (I+1), BufSize-(I+1)) + Skipped; + FilePos -= GetStrBytesNum(tBuf + (I+1), BufSize-(I+1)) + Skipped; return; } else @@ -2003,7 +2007,7 @@ if (CalcStrSize(&Buf[J],BufSize-J) <= Width) { - FilePos -= GetStrBytesNum(Buf + J, BufSize-J) + Skipped; + FilePos -= GetStrBytesNum(tBuf + J, BufSize-J) + Skipped; return; } else @@ -2025,7 +2029,7 @@ for (I=Min(Width,BufSize); I>0; I-=5) if (CalcStrSize(&Buf[BufSize-I],I) <= Width) { - FilePos -= GetStrBytesNum(&Buf[BufSize-I], I)+Skipped; + FilePos -= GetStrBytesNum(&tBuf[BufSize-I], I)+Skipped; break; } } @@ -2309,6 +2313,7 @@ ReverseSearch=SearchDlg[SD_CHECKBOX_REVERSE].Selected; SearchRegexp=SearchDlg[SD_CHECKBOX_REGEXP].Selected; + bool t_utf8 = false; if (SearchHex) { strSearchStr = SearchDlg[SD_EDIT_HEX].strData; @@ -2317,6 +2322,7 @@ else { strSearchStr = SearchDlg[SD_EDIT_TEXT].strData; + t_utf8 = VM.CodePage == CP_UTF8; } strLastSearchStr = strSearchStr; @@ -2380,7 +2386,7 @@ if (SearchLength>0 && (!ReverseSearch || LastSelPos>0)) { - wchar_t Buf[8192]; + wchar_t Buf[8192], t_Buf[ARRAYSIZE(Buf)]; __int64 CurPos=LastSelPos; int BufSize=ARRAYSIZE(Buf); @@ -2416,10 +2422,14 @@ vseek(CurPos,SEEK_SET); - if ((ReadSize=vread(Buf,BufSize,SearchHex!=0))<=0) - break; + ReadSize = t_utf8 + ? vread(Buf, BufSize, false, t_Buf) + : vread(Buf, BufSize, SearchHex!=0) + ; + if (ReadSize < 0) + break; - DWORD CurTime=GetTickCount(); + DWORD CurTime=GetTickCount(); if (CurTime-StartTime>RedrawTimeout) { @@ -2508,6 +2518,11 @@ if (Match) { MatchPos=CurPos+I; + if (t_utf8) + { + MatchPos = CurPos + GetStrBytesNum(t_Buf, I); + SearchLength = GetStrBytesNum(t_Buf+I, SearchLength); + } break; } } @@ -2705,9 +2720,88 @@ List->MoveData(ViewNamesList); } +#define REPLACEMENT 0xFFFD // L'?' -int Viewer::vread(wchar_t *Buf,int Count, bool Raw) +static int utf8_to_WidwChar( + const char *s, int nc, wchar_t *w1,wchar_t *w2, int wlen, int &tail ) { + if ( wlen <= 0 ) + { tail = nc; return 0; } + + int ic = 0, nw = 0, wc; + while ( ic < nc ) + { + unsigned char c4, c3, c2, c1 = ((const unsigned char *)s)[ic++]; + + if (c1 < 0x80) // simple ASCII + wc = (wchar_t)c1; + else if ( c1 < 0xC2 || c1 > 0xF5 ) // illegal 1-st byte + wc = -1; + else { // multibyte (2, 3, 4) + if (ic + 0 >= nc ) { // unfinished +unfinished: if ( nw > 0 ) tail = nc - ic + 1; + else { tail = 0; w1[0] = REPLACEMENT; if (w2) w2[0] = L'?'; nw = 1; } + return nw; + } + c2 = ((const unsigned char *)s)[ic]; + if ( 0x80 != (c2 & 0xC0) // illegal 2-st byte + || (0xE0 == c1 && c2 <= 0x9F) // illegal 3-byte start (overlaps with 2-byte) + || (0xF0 == c1 && c2 <= 0x8F) // illegal 4-byte start (overlaps with 3-byte) + || (0xF4 == c1 && c2 >= 0x90) // illegal 4-byte (out of unicode range) + ) { + wc = -1; + } + else if ( c1 < 0xE0 ) { // legal 2-byte + ++ic; + wc = ((c1 & 0x1F) << 6) | (c2 & 0x3F); + } + else { // 3 or 4-byte + if (ic + 1 >= nc ) goto unfinished; + c3 = ((const unsigned char *)s)[ic+1]; + if ( 0x80 != (c3 & 0xC0) ) // illegal 3-rd byte + wc = -1; + else if ( c1 < 0xF0 ) { // legal 3-byte + ic += 2; + wc = ((c1 & 0x0F) << 12) | ((c2 & 0x3F) << 6) | (c3 & 0x3F); + } + else { // 4-byte + if (ic + 2 >= nc ) goto unfinished; + c4 = ((const unsigned char *)s)[ic+2]; + if ( 0x80 != (c4 & 0xC0) ) // illegal 4-th byte + wc = -1; + else { // legal 4-byte (produce 2 WCHARs) + ic += 3; + wc = ((c1 & 0x07) << 18) | ((c2 & 0x3F) << 12) | ((c3 & 0x3F) << 6) | (c4 & 0x3F); + wc -= 0x10000; + w1[nw] = (wchar_t)(0xD800 + (wc >> 10)); + if (w2) w2[nw] = w1[nw]; + ++nw; + wc = 0xDC00 + (wc & 0x3FF); +#if 0 + _ASSERTE(nw < wlen); //??? caller should be fixed to avoid this... +#endif + if (nw >= wlen) { --nw; wc = REPLACEMENT; } + } + } + } + } + + if ( wc >= 0 ) { + w1[nw] = (wchar_t)wc; if (w2) w2[nw] = (wchar_t)wc; + } + else { + w1[nw] = REPLACEMENT; if (w2) w2[nw] = L'?'; + } + if (++nw >= wlen) + break; + } + + tail = nc - ic; + return nw; +} + +int Viewer::vread(wchar_t *Buf,int Count, bool Raw, wchar_t *Buf2) +{ if (IsUnicodeCodePage(VM.CodePage)) { // ������ ������, ������ �����! @@ -2765,7 +2859,8 @@ 11110zzz 10zzyyyy 10yyyyxx 10xxxxxx */ - //� ������� ����? ����� ��� ����� �������. +#if 0 + //� ������� ����? ����� ��� ����� �������. while ((c&0xC0) == 0x80) { DWORD Readed=0; @@ -2775,6 +2870,7 @@ ReadSize++; c=*TmpBuf; } +#endif //�������� ������ ���� ��� � ���� ������� ���� ������ ���� �����. DWORD cc=1; @@ -2801,7 +2897,15 @@ } else { - ReadSize = MultiByteToWideChar(VM.CodePage, 0, TmpBuf, ConvertSize, Buf, Count); + if (VM.CodePage != CP_UTF8) + ReadSize = MultiByteToWideChar(VM.CodePage, 0, TmpBuf, ConvertSize, Buf, Count); + else + { + int tail; + ReadSize = (DWORD)utf8_to_WidwChar(TmpBuf, ConvertSize, Buf,Buf2, Count, tail); + if (tail) + Reader.Unread(tail); + } } xf_free(TmpBuf); @@ -2812,6 +2916,7 @@ int Viewer::vseek(__int64 Offset,int Whence) { + vgetc_ready = -1; return ViewFile.SetPointer(Offset*(IsUnicodeCodePage(VM.CodePage)?2:1), nullptr, Whence); } @@ -2830,12 +2935,29 @@ bool Viewer::vgetc(WCHAR& C) { - bool Result=false; - - if (vread(&C,1)) - { - Result=true; - } + bool Result; + if (VM.Hex || VM.CodePage != CP_UTF8) + Result = (vread(&C,1) > 0); + else { + if (vgetc_ready >= 0) { + C = (WCHAR)vgetc_ready; vgetc_ready = -1; Result = true; + } + else { + wchar_t w1[4], w2[4]; + int need = 1; + int nw = vread(w1, 4, false, w2); + if (true == (Result = (nw > 0))) { + C = w1[0]; + if (nw > need && 0xD800 == (w1[0] & 0xFC00) && 0xDC00 == (w1[1] & 0xFC00)) { + ++need; vgetc_ready = w1[1]; + } + if (nw > need) { + int tail = GetStrBytesNum(w2+need, nw-need); + Reader.Unread(tail); + } + } + } + } return Result; } |
|
viewer.diff - мои изменения (cache/viewer .cpp/.hpp) По крайней мере делает вид, что работает... |
|
viewer_fixed.diff (9,731 bytes)
Index: cache.hpp =================================================================== --- cache.hpp (revision 5769) +++ cache.hpp (working copy) @@ -39,6 +39,7 @@ ~CachedRead(); bool Read(LPVOID Data, DWORD DataSize, LPDWORD BytesRead); bool FillBuffer(); + bool Unread(DWORD BytesUnread); void Clear(); private: Index: viewer.hpp =================================================================== --- viewer.hpp (revision 5769) +++ viewer.hpp (working copy) @@ -163,6 +163,9 @@ bool m_bQuickView; UINT DefCodePage; + + int vgetc_ready; + private: virtual void DisplayObject(); @@ -187,7 +190,7 @@ void Search(int Next,int FirstChar); void ConvertToHex(char *SearchStr,int &SearchLength); int HexToNum(int Hex); - int vread(wchar_t *Buf,int Count, bool Raw=false); + int vread(wchar_t *Buf,int Count, bool Raw=false, wchar_t *Buf2=NULL); int vseek(__int64 Offset,int Whence); __int64 vtell(); bool vgetc(WCHAR& C); Index: cache.cpp =================================================================== --- cache.cpp (revision 5769) +++ cache.cpp (working copy) @@ -109,6 +109,18 @@ return Result; } +bool CachedRead::Unread(DWORD BytesUnread) +{ + if (BytesUnread + BytesLeft <= ReadSize) + { + BytesLeft += BytesUnread; + __int64 off = BytesUnread; + file.SetPointer(-off, &LastPtr, FILE_CURRENT); + return true; + } + return false; +} + bool CachedRead::FillBuffer() { bool Result=false; Index: viewer.cpp =================================================================== --- viewer.cpp (revision 5769) +++ viewer.cpp (working copy) @@ -122,6 +122,7 @@ SelectPosOffSet=0; bVE_READ_Sent = false; Signature = false; + vgetc_ready = -1; } @@ -1929,9 +1930,12 @@ if (!ViewFile.Opened()) return; - wchar_t Buf[MAX_VIEWLINE]; + wchar_t Buf[MAX_VIEWLINE], Buf2[ARRAYSIZE(Buf)], *tBuf = Buf, *tBuf2 = NULL; int BufSize,StrPos,Skipped,I,J; + if (!VM.Hex && VM.CodePage == CP_UTF8) + tBuf = tBuf2 = Buf2; + if (FilePos > (__int64)sizeof(Buf)/sizeof(wchar_t)) BufSize=sizeof(Buf)/sizeof(wchar_t); else @@ -1955,7 +1959,7 @@ } vseek(FilePos-(__int64)BufSize,SEEK_SET); - BufSize = vread(Buf,BufSize); + BufSize = vread(Buf, BufSize, false, tBuf2); Skipped=0; if (BufSize>0 && Buf[BufSize-1]==(unsigned int)CRSym) @@ -1979,7 +1983,7 @@ { if (!VM.Wrap) { - FilePos -= GetStrBytesNum(Buf + (I+1), BufSize-(I+1)) + Skipped; + FilePos -= GetStrBytesNum(tBuf + (I+1), BufSize-(I+1)) + Skipped; return; } else @@ -2003,7 +2007,7 @@ if (CalcStrSize(&Buf[J],BufSize-J) <= Width) { - FilePos -= GetStrBytesNum(Buf + J, BufSize-J) + Skipped; + FilePos -= GetStrBytesNum(tBuf + J, BufSize-J) + Skipped; return; } else @@ -2025,7 +2029,7 @@ for (I=Min(Width,BufSize); I>0; I-=5) if (CalcStrSize(&Buf[BufSize-I],I) <= Width) { - FilePos -= GetStrBytesNum(&Buf[BufSize-I], I)+Skipped; + FilePos -= GetStrBytesNum(&tBuf[BufSize-I], I)+Skipped; break; } } @@ -2309,6 +2313,7 @@ ReverseSearch=SearchDlg[SD_CHECKBOX_REVERSE].Selected; SearchRegexp=SearchDlg[SD_CHECKBOX_REGEXP].Selected; + bool t_utf8 = false; if (SearchHex) { strSearchStr = SearchDlg[SD_EDIT_HEX].strData; @@ -2317,6 +2322,7 @@ else { strSearchStr = SearchDlg[SD_EDIT_TEXT].strData; + t_utf8 = VM.CodePage == CP_UTF8; } strLastSearchStr = strSearchStr; @@ -2380,7 +2386,7 @@ if (SearchLength>0 && (!ReverseSearch || LastSelPos>0)) { - wchar_t Buf[8192]; + wchar_t Buf[8192], t_Buf[ARRAYSIZE(Buf)]; __int64 CurPos=LastSelPos; int BufSize=ARRAYSIZE(Buf); @@ -2416,10 +2422,14 @@ vseek(CurPos,SEEK_SET); - if ((ReadSize=vread(Buf,BufSize,SearchHex!=0))<=0) - break; + ReadSize = t_utf8 + ? vread(Buf, BufSize, false, t_Buf) + : vread(Buf, BufSize, SearchHex!=0) + ; + if (ReadSize <= 0) + break; - DWORD CurTime=GetTickCount(); + DWORD CurTime=GetTickCount(); if (CurTime-StartTime>RedrawTimeout) { @@ -2508,6 +2518,11 @@ if (Match) { MatchPos=CurPos+I; + if (t_utf8) + { + MatchPos = CurPos + GetStrBytesNum(t_Buf, I); + SearchLength = GetStrBytesNum(t_Buf+I, SearchLength); + } break; } } @@ -2705,9 +2720,88 @@ List->MoveData(ViewNamesList); } +#define REPLACEMENT 0xFFFD // L'?' -int Viewer::vread(wchar_t *Buf,int Count, bool Raw) +static int utf8_to_WidwChar( + const char *s, int nc, wchar_t *w1,wchar_t *w2, int wlen, int &tail ) { + if ( wlen <= 0 ) + { tail = nc; return 0; } + + int ic = 0, nw = 0, wc; + while ( ic < nc ) + { + unsigned char c4, c3, c2, c1 = ((const unsigned char *)s)[ic++]; + + if (c1 < 0x80) // simple ASCII + wc = (wchar_t)c1; + else if ( c1 < 0xC2 || c1 > 0xF5 ) // illegal 1-st byte + wc = -1; + else { // multibyte (2, 3, 4) + if (ic + 0 >= nc ) { // unfinished +unfinished: if ( nw > 0 ) tail = nc - ic + 1; + else { tail = 0; w1[0] = REPLACEMENT; if (w2) w2[0] = L'?'; nw = 1; } + return nw; + } + c2 = ((const unsigned char *)s)[ic]; + if ( 0x80 != (c2 & 0xC0) // illegal 2-st byte + || (0xE0 == c1 && c2 <= 0x9F) // illegal 3-byte start (overlaps with 2-byte) + || (0xF0 == c1 && c2 <= 0x8F) // illegal 4-byte start (overlaps with 3-byte) + || (0xF4 == c1 && c2 >= 0x90) // illegal 4-byte (out of unicode range) + ) { + wc = -1; + } + else if ( c1 < 0xE0 ) { // legal 2-byte + ++ic; + wc = ((c1 & 0x1F) << 6) | (c2 & 0x3F); + } + else { // 3 or 4-byte + if (ic + 1 >= nc ) goto unfinished; + c3 = ((const unsigned char *)s)[ic+1]; + if ( 0x80 != (c3 & 0xC0) ) // illegal 3-rd byte + wc = -1; + else if ( c1 < 0xF0 ) { // legal 3-byte + ic += 2; + wc = ((c1 & 0x0F) << 12) | ((c2 & 0x3F) << 6) | (c3 & 0x3F); + } + else { // 4-byte + if (ic + 2 >= nc ) goto unfinished; + c4 = ((const unsigned char *)s)[ic+2]; + if ( 0x80 != (c4 & 0xC0) ) // illegal 4-th byte + wc = -1; + else { // legal 4-byte (produce 2 WCHARs) + ic += 3; + wc = ((c1 & 0x07) << 18) | ((c2 & 0x3F) << 12) | ((c3 & 0x3F) << 6) | (c4 & 0x3F); + wc -= 0x10000; + w1[nw] = (wchar_t)(0xD800 + (wc >> 10)); + if (w2) w2[nw] = w1[nw]; + ++nw; + wc = 0xDC00 + (wc & 0x3FF); +#if 0 + _ASSERTE(nw < wlen); //??? caller should be fixed to avoid this... +#endif + if (nw >= wlen) { --nw; wc = REPLACEMENT; } + } + } + } + } + + if ( wc >= 0 ) { + w1[nw] = (wchar_t)wc; if (w2) w2[nw] = (wchar_t)wc; + } + else { + w1[nw] = REPLACEMENT; if (w2) w2[nw] = L'?'; + } + if (++nw >= wlen) + break; + } + + tail = nc - ic; + return nw; +} + +int Viewer::vread(wchar_t *Buf,int Count, bool Raw, wchar_t *Buf2) +{ if (IsUnicodeCodePage(VM.CodePage)) { // ������ ������, ������ �����! @@ -2765,7 +2859,8 @@ 11110zzz 10zzyyyy 10yyyyxx 10xxxxxx */ - //� ������� ����? ����� ��� ����� �������. +#if 0 + //� ������� ����? ����� ��� ����� �������. while ((c&0xC0) == 0x80) { DWORD Readed=0; @@ -2775,6 +2870,7 @@ ReadSize++; c=*TmpBuf; } +#endif //�������� ������ ���� ��� � ���� ������� ���� ������ ���� �����. DWORD cc=1; @@ -2801,7 +2897,15 @@ } else { - ReadSize = MultiByteToWideChar(VM.CodePage, 0, TmpBuf, ConvertSize, Buf, Count); + if (VM.CodePage != CP_UTF8) + ReadSize = MultiByteToWideChar(VM.CodePage, 0, TmpBuf, ConvertSize, Buf, Count); + else + { + int tail; + ReadSize = (DWORD)utf8_to_WidwChar(TmpBuf, ConvertSize, Buf,Buf2, Count, tail); + if (tail) + Reader.Unread(tail); + } } xf_free(TmpBuf); @@ -2812,6 +2916,7 @@ int Viewer::vseek(__int64 Offset,int Whence) { + vgetc_ready = -1; return ViewFile.SetPointer(Offset*(IsUnicodeCodePage(VM.CodePage)?2:1), nullptr, Whence); } @@ -2830,12 +2935,29 @@ bool Viewer::vgetc(WCHAR& C) { - bool Result=false; - - if (vread(&C,1)) - { - Result=true; - } + bool Result; + if (VM.Hex || VM.CodePage != CP_UTF8) + Result = (vread(&C,1) > 0); + else { + if (vgetc_ready >= 0) { + C = (WCHAR)vgetc_ready; vgetc_ready = -1; Result = true; + } + else { + wchar_t w1[4], w2[4]; + int need = 1; + int nw = vread(w1, 4, false, w2); + if (true == (Result = (nw > 0))) { + C = w1[0]; + if (nw > need && 0xD800 == (w1[0] & 0xFC00) && 0xDC00 == (w1[1] & 0xFC00)) { + ++need; vgetc_ready = w1[1]; + } + if (nw > need) { + int tail = GetStrBytesNum(w2+need, nw-need); + Reader.Unread(tail); + } + } + } + } return Result; } |
|
viewer_fixed2.diff (9,732 bytes)
Index: cache.hpp =================================================================== --- cache.hpp (revision 5769) +++ cache.hpp (working copy) @@ -39,6 +39,7 @@ ~CachedRead(); bool Read(LPVOID Data, DWORD DataSize, LPDWORD BytesRead); bool FillBuffer(); + bool Unread(DWORD BytesUnread); void Clear(); private: Index: viewer.hpp =================================================================== --- viewer.hpp (revision 5769) +++ viewer.hpp (working copy) @@ -163,6 +163,9 @@ bool m_bQuickView; UINT DefCodePage; + + int vgetc_ready; + private: virtual void DisplayObject(); @@ -187,7 +190,7 @@ void Search(int Next,int FirstChar); void ConvertToHex(char *SearchStr,int &SearchLength); int HexToNum(int Hex); - int vread(wchar_t *Buf,int Count, bool Raw=false); + int vread(wchar_t *Buf,int Count, bool Raw=false, wchar_t *Buf2=NULL); int vseek(__int64 Offset,int Whence); __int64 vtell(); bool vgetc(WCHAR& C); Index: cache.cpp =================================================================== --- cache.cpp (revision 5769) +++ cache.cpp (working copy) @@ -109,6 +109,18 @@ return Result; } +bool CachedRead::Unread(DWORD BytesUnread) +{ + if (BytesUnread + BytesLeft <= ReadSize) + { + BytesLeft += BytesUnread; + __int64 off = BytesUnread; + file.SetPointer(-off, &LastPtr, FILE_CURRENT); + return true; + } + return false; +} + bool CachedRead::FillBuffer() { bool Result=false; Index: viewer.cpp =================================================================== --- viewer.cpp (revision 5769) +++ viewer.cpp (working copy) @@ -122,6 +122,7 @@ SelectPosOffSet=0; bVE_READ_Sent = false; Signature = false; + vgetc_ready = -1; } @@ -1929,9 +1930,12 @@ if (!ViewFile.Opened()) return; - wchar_t Buf[MAX_VIEWLINE]; + wchar_t Buf[MAX_VIEWLINE], Buf2[ARRAYSIZE(Buf)], *tBuf = Buf, *tBuf2 = NULL; int BufSize,StrPos,Skipped,I,J; + if (!VM.Hex && VM.CodePage == CP_UTF8) + tBuf = tBuf2 = Buf2; + if (FilePos > (__int64)sizeof(Buf)/sizeof(wchar_t)) BufSize=sizeof(Buf)/sizeof(wchar_t); else @@ -1955,7 +1959,7 @@ } vseek(FilePos-(__int64)BufSize,SEEK_SET); - BufSize = vread(Buf,BufSize); + BufSize = vread(Buf, BufSize, false, tBuf2); Skipped=0; if (BufSize>0 && Buf[BufSize-1]==(unsigned int)CRSym) @@ -1979,7 +1983,7 @@ { if (!VM.Wrap) { - FilePos -= GetStrBytesNum(Buf + (I+1), BufSize-(I+1)) + Skipped; + FilePos -= GetStrBytesNum(tBuf + (I+1), BufSize-(I+1)) + Skipped; return; } else @@ -2003,7 +2007,7 @@ if (CalcStrSize(&Buf[J],BufSize-J) <= Width) { - FilePos -= GetStrBytesNum(Buf + J, BufSize-J) + Skipped; + FilePos -= GetStrBytesNum(tBuf + J, BufSize-J) + Skipped; return; } else @@ -2025,7 +2029,7 @@ for (I=Min(Width,BufSize); I>0; I-=5) if (CalcStrSize(&Buf[BufSize-I],I) <= Width) { - FilePos -= GetStrBytesNum(&Buf[BufSize-I], I)+Skipped; + FilePos -= GetStrBytesNum(&tBuf[BufSize-I], I)+Skipped; break; } } @@ -2309,6 +2313,7 @@ ReverseSearch=SearchDlg[SD_CHECKBOX_REVERSE].Selected; SearchRegexp=SearchDlg[SD_CHECKBOX_REGEXP].Selected; + bool t_utf8 = false; if (SearchHex) { strSearchStr = SearchDlg[SD_EDIT_HEX].strData; @@ -2317,6 +2322,7 @@ else { strSearchStr = SearchDlg[SD_EDIT_TEXT].strData; + t_utf8 = VM.CodePage == CP_UTF8; } strLastSearchStr = strSearchStr; @@ -2380,7 +2386,7 @@ if (SearchLength>0 && (!ReverseSearch || LastSelPos>0)) { - wchar_t Buf[8192]; + wchar_t Buf[8192], t_Buf[ARRAYSIZE(Buf)]; __int64 CurPos=LastSelPos; int BufSize=ARRAYSIZE(Buf); @@ -2416,10 +2422,14 @@ vseek(CurPos,SEEK_SET); - if ((ReadSize=vread(Buf,BufSize,SearchHex!=0))<=0) - break; + ReadSize = t_utf8 + ? vread(Buf, BufSize, false, t_Buf) + : vread(Buf, BufSize, SearchHex!=0) + ; + if (ReadSize <= 0) + break; - DWORD CurTime=GetTickCount(); + DWORD CurTime=GetTickCount(); if (CurTime-StartTime>RedrawTimeout) { @@ -2508,6 +2518,11 @@ if (Match) { MatchPos=CurPos+I; + if (t_utf8) + { + MatchPos = CurPos + GetStrBytesNum(t_Buf, I); + SearchLength = GetStrBytesNum(t_Buf+I, SearchLength); + } break; } } @@ -2705,9 +2720,88 @@ List->MoveData(ViewNamesList); } +#define REPLACEMENT 0xFFFD // L'?' -int Viewer::vread(wchar_t *Buf,int Count, bool Raw) +static int utf8_to_WideChar( + const char *s, int nc, wchar_t *w1,wchar_t *w2, int wlen, int &tail ) { + if ( wlen <= 0 ) + { tail = nc; return 0; } + + int ic = 0, nw = 0, wc; + while ( ic < nc ) + { + unsigned char c4, c3, c2, c1 = ((const unsigned char *)s)[ic++]; + + if (c1 < 0x80) // simple ASCII + wc = (wchar_t)c1; + else if ( c1 < 0xC2 || c1 >= 0xF5 ) // illegal 1-st byte + wc = -1; + else { // multibyte (2, 3, 4) + if (ic + 0 >= nc ) { // unfinished +unfinished: if ( nw > 0 ) tail = nc - ic + 1; + else { tail = 0; w1[0] = REPLACEMENT; if (w2) w2[0] = L'?'; nw = 1; } + return nw; + } + c2 = ((const unsigned char *)s)[ic]; + if ( 0x80 != (c2 & 0xC0) // illegal 2-st byte + || (0xE0 == c1 && c2 <= 0x9F) // illegal 3-byte start (overlaps with 2-byte) + || (0xF0 == c1 && c2 <= 0x8F) // illegal 4-byte start (overlaps with 3-byte) + || (0xF4 == c1 && c2 >= 0x90) // illegal 4-byte (out of unicode range) + ) { + wc = -1; + } + else if ( c1 < 0xE0 ) { // legal 2-byte + ++ic; + wc = ((c1 & 0x1F) << 6) | (c2 & 0x3F); + } + else { // 3 or 4-byte + if (ic + 1 >= nc ) goto unfinished; + c3 = ((const unsigned char *)s)[ic+1]; + if ( 0x80 != (c3 & 0xC0) ) // illegal 3-rd byte + wc = -1; + else if ( c1 < 0xF0 ) { // legal 3-byte + ic += 2; + wc = ((c1 & 0x0F) << 12) | ((c2 & 0x3F) << 6) | (c3 & 0x3F); + } + else { // 4-byte + if (ic + 2 >= nc ) goto unfinished; + c4 = ((const unsigned char *)s)[ic+2]; + if ( 0x80 != (c4 & 0xC0) ) // illegal 4-th byte + wc = -1; + else { // legal 4-byte (produce 2 WCHARs) + ic += 3; + wc = ((c1 & 0x07) << 18) | ((c2 & 0x3F) << 12) | ((c3 & 0x3F) << 6) | (c4 & 0x3F); + wc -= 0x10000; + w1[nw] = (wchar_t)(0xD800 + (wc >> 10)); + if (w2) w2[nw] = w1[nw]; + ++nw; + wc = 0xDC00 + (wc & 0x3FF); +#if 0 + _ASSERTE(nw < wlen); //??? caller should be fixed to avoid this... +#endif + if (nw >= wlen) { --nw; wc = REPLACEMENT; } + } + } + } + } + + if ( wc >= 0 ) { + w1[nw] = (wchar_t)wc; if (w2) w2[nw] = (wchar_t)wc; + } + else { + w1[nw] = REPLACEMENT; if (w2) w2[nw] = L'?'; + } + if (++nw >= wlen) + break; + } + + tail = nc - ic; + return nw; +} + +int Viewer::vread(wchar_t *Buf,int Count, bool Raw, wchar_t *Buf2) +{ if (IsUnicodeCodePage(VM.CodePage)) { // ������ ������, ������ �����! @@ -2765,7 +2859,8 @@ 11110zzz 10zzyyyy 10yyyyxx 10xxxxxx */ - //� ������� ����? ����� ��� ����� �������. +#if 0 + //� ������� ����? ����� ��� ����� �������. while ((c&0xC0) == 0x80) { DWORD Readed=0; @@ -2775,6 +2870,7 @@ ReadSize++; c=*TmpBuf; } +#endif //�������� ������ ���� ��� � ���� ������� ���� ������ ���� �����. DWORD cc=1; @@ -2801,7 +2897,15 @@ } else { - ReadSize = MultiByteToWideChar(VM.CodePage, 0, TmpBuf, ConvertSize, Buf, Count); + if (VM.CodePage != CP_UTF8) + ReadSize = MultiByteToWideChar(VM.CodePage, 0, TmpBuf, ConvertSize, Buf, Count); + else + { + int tail; + ReadSize = (DWORD)utf8_to_WideChar(TmpBuf, ConvertSize, Buf,Buf2, Count, tail); + if (tail) + Reader.Unread(tail); + } } xf_free(TmpBuf); @@ -2812,6 +2916,7 @@ int Viewer::vseek(__int64 Offset,int Whence) { + vgetc_ready = -1; return ViewFile.SetPointer(Offset*(IsUnicodeCodePage(VM.CodePage)?2:1), nullptr, Whence); } @@ -2830,12 +2935,29 @@ bool Viewer::vgetc(WCHAR& C) { - bool Result=false; - - if (vread(&C,1)) - { - Result=true; - } + bool Result; + if (VM.Hex || VM.CodePage != CP_UTF8) + Result = (vread(&C,1) > 0); + else { + if (vgetc_ready >= 0) { + C = (WCHAR)vgetc_ready; vgetc_ready = -1; Result = true; + } + else { + wchar_t w1[4], w2[4]; + int need = 1; + int nw = vread(w1, 4, false, w2); + if (true == (Result = (nw > 0))) { + C = w1[0]; + if (nw > need && 0xD800 == (w1[0] & 0xFC00) && 0xDC00 == (w1[1] & 0xFC00)) { + ++need; vgetc_ready = w1[1]; + } + if (nw > need) { + int tail = GetStrBytesNum(w2+need, nw-need); + Reader.Unread(tail); + } + } + } + } return Result; } |
|
чей код функции utf8_to_WidwChar? откуда взят? какие на него права/лицензии? etc... |
|
код мой. absolutly free... опечатку в WidwChar я уже исправил. |
|
А чем MultiByteToWideChar(CP_UTF8) не подошёл? |
|
До висты в MultiByteToWideChar() некорректные последовательности просто игнорируются. К тому же у меня делается два варианта перекодировки: один с replacement (0xfffd) для показа, другой с '?' для пересчета смещения (замена должна быть одним байтом в utf-8). |
|
хм, очень похоже на то что надо. надо закомитить. |
|
рановато коммитить... что-то у меня с поиском не так - разберусь, дам знать |
|
открываю во вьювере FarRus.hlf.m4 Home f7 индекс Enter блаблабла в общем только 2 раза находит... |
|
срабатывает условие: if ((ReverseSearch && CurPos <= 0) || (!ReverseSearch && ReadSize < BufSize)) break; пофиксить думаю будет несложно... Например так: if ((ReverseSearch && CurPos <= 0) || (!ReverseSearch && ViewFile.Eof())) break; |
|
viewer_fixed3.diff (9,950 bytes)
Index: cache.hpp =================================================================== --- cache.hpp (revision 5769) +++ cache.hpp (working copy) @@ -39,6 +39,7 @@ ~CachedRead(); bool Read(LPVOID Data, DWORD DataSize, LPDWORD BytesRead); bool FillBuffer(); + bool Unread(DWORD BytesUnread); void Clear(); private: Index: viewer.hpp =================================================================== --- viewer.hpp (revision 5769) +++ viewer.hpp (working copy) @@ -163,6 +163,9 @@ bool m_bQuickView; UINT DefCodePage; + + int vgetc_ready; + private: virtual void DisplayObject(); @@ -187,7 +190,7 @@ void Search(int Next,int FirstChar); void ConvertToHex(char *SearchStr,int &SearchLength); int HexToNum(int Hex); - int vread(wchar_t *Buf,int Count, bool Raw=false); + int vread(wchar_t *Buf,int Count, bool Raw=false, wchar_t *Buf2=NULL); int vseek(__int64 Offset,int Whence); __int64 vtell(); bool vgetc(WCHAR& C); Index: cache.cpp =================================================================== --- cache.cpp (revision 5769) +++ cache.cpp (working copy) @@ -109,6 +109,18 @@ return Result; } +bool CachedRead::Unread(DWORD BytesUnread) +{ + if (BytesUnread + BytesLeft <= ReadSize) + { + BytesLeft += BytesUnread; + __int64 off = BytesUnread; + file.SetPointer(-off, &LastPtr, FILE_CURRENT); + return true; + } + return false; +} + bool CachedRead::FillBuffer() { bool Result=false; Index: viewer.cpp =================================================================== --- viewer.cpp (revision 5769) +++ viewer.cpp (working copy) @@ -122,6 +122,7 @@ SelectPosOffSet=0; bVE_READ_Sent = false; Signature = false; + vgetc_ready = -1; } @@ -1929,9 +1930,12 @@ if (!ViewFile.Opened()) return; - wchar_t Buf[MAX_VIEWLINE]; + wchar_t Buf[MAX_VIEWLINE], Buf2[ARRAYSIZE(Buf)], *tBuf = Buf, *tBuf2 = NULL; int BufSize,StrPos,Skipped,I,J; + if (!VM.Hex && VM.CodePage == CP_UTF8) + tBuf = tBuf2 = Buf2; + if (FilePos > (__int64)sizeof(Buf)/sizeof(wchar_t)) BufSize=sizeof(Buf)/sizeof(wchar_t); else @@ -1955,7 +1959,7 @@ } vseek(FilePos-(__int64)BufSize,SEEK_SET); - BufSize = vread(Buf,BufSize); + BufSize = vread(Buf, BufSize, false, tBuf2); Skipped=0; if (BufSize>0 && Buf[BufSize-1]==(unsigned int)CRSym) @@ -1979,7 +1983,7 @@ { if (!VM.Wrap) { - FilePos -= GetStrBytesNum(Buf + (I+1), BufSize-(I+1)) + Skipped; + FilePos -= GetStrBytesNum(tBuf + (I+1), BufSize-(I+1)) + Skipped; return; } else @@ -2003,7 +2007,7 @@ if (CalcStrSize(&Buf[J],BufSize-J) <= Width) { - FilePos -= GetStrBytesNum(Buf + J, BufSize-J) + Skipped; + FilePos -= GetStrBytesNum(tBuf + J, BufSize-J) + Skipped; return; } else @@ -2025,7 +2029,7 @@ for (I=Min(Width,BufSize); I>0; I-=5) if (CalcStrSize(&Buf[BufSize-I],I) <= Width) { - FilePos -= GetStrBytesNum(&Buf[BufSize-I], I)+Skipped; + FilePos -= GetStrBytesNum(&tBuf[BufSize-I], I)+Skipped; break; } } @@ -2309,6 +2313,7 @@ ReverseSearch=SearchDlg[SD_CHECKBOX_REVERSE].Selected; SearchRegexp=SearchDlg[SD_CHECKBOX_REGEXP].Selected; + bool t_utf8 = false; if (SearchHex) { strSearchStr = SearchDlg[SD_EDIT_HEX].strData; @@ -2317,6 +2322,7 @@ else { strSearchStr = SearchDlg[SD_EDIT_TEXT].strData; + t_utf8 = VM.CodePage == CP_UTF8; } strLastSearchStr = strSearchStr; @@ -2380,7 +2386,7 @@ if (SearchLength>0 && (!ReverseSearch || LastSelPos>0)) { - wchar_t Buf[8192]; + wchar_t Buf[8192], t_Buf[ARRAYSIZE(Buf)]; __int64 CurPos=LastSelPos; int BufSize=ARRAYSIZE(Buf); @@ -2416,10 +2422,14 @@ vseek(CurPos,SEEK_SET); - if ((ReadSize=vread(Buf,BufSize,SearchHex!=0))<=0) - break; + ReadSize = t_utf8 + ? vread(Buf, BufSize, false, t_Buf) + : vread(Buf, BufSize, SearchHex!=0) + ; + if (ReadSize <= 0) + break; - DWORD CurTime=GetTickCount(); + DWORD CurTime=GetTickCount(); if (CurTime-StartTime>RedrawTimeout) { @@ -2508,11 +2518,16 @@ if (Match) { MatchPos=CurPos+I; + if (t_utf8) + { + MatchPos = CurPos + GetStrBytesNum(t_Buf, I); + SearchLength = GetStrBytesNum(t_Buf+I, SearchLength); + } break; } } - if ((ReverseSearch && CurPos <= 0) || (!ReverseSearch && ReadSize < BufSize)) + if ((ReverseSearch && CurPos <= 0) || (!ReverseSearch && ViewFile.Eof())) break; if (ReverseSearch) @@ -2705,9 +2720,88 @@ List->MoveData(ViewNamesList); } +#define REPLACEMENT 0xFFFD // L'?' -int Viewer::vread(wchar_t *Buf,int Count, bool Raw) +static int utf8_to_WideChar( + const char *s, int nc, wchar_t *w1,wchar_t *w2, int wlen, int &tail ) { + if ( wlen <= 0 ) + { tail = nc; return 0; } + + int ic = 0, nw = 0, wc; + while ( ic < nc ) + { + unsigned char c4, c3, c2, c1 = ((const unsigned char *)s)[ic++]; + + if (c1 < 0x80) // simple ASCII + wc = (wchar_t)c1; + else if ( c1 < 0xC2 || c1 >= 0xF5 ) // illegal 1-st byte + wc = -1; + else { // multibyte (2, 3, 4) + if (ic + 0 >= nc ) { // unfinished +unfinished: if ( nw > 0 ) tail = nc - ic + 1; + else { tail = 0; w1[0] = REPLACEMENT; if (w2) w2[0] = L'?'; nw = 1; } + return nw; + } + c2 = ((const unsigned char *)s)[ic]; + if ( 0x80 != (c2 & 0xC0) // illegal 2-st byte + || (0xE0 == c1 && c2 <= 0x9F) // illegal 3-byte start (overlaps with 2-byte) + || (0xF0 == c1 && c2 <= 0x8F) // illegal 4-byte start (overlaps with 3-byte) + || (0xF4 == c1 && c2 >= 0x90) // illegal 4-byte (out of unicode range) + ) { + wc = -1; + } + else if ( c1 < 0xE0 ) { // legal 2-byte + ++ic; + wc = ((c1 & 0x1F) << 6) | (c2 & 0x3F); + } + else { // 3 or 4-byte + if (ic + 1 >= nc ) goto unfinished; + c3 = ((const unsigned char *)s)[ic+1]; + if ( 0x80 != (c3 & 0xC0) ) // illegal 3-rd byte + wc = -1; + else if ( c1 < 0xF0 ) { // legal 3-byte + ic += 2; + wc = ((c1 & 0x0F) << 12) | ((c2 & 0x3F) << 6) | (c3 & 0x3F); + } + else { // 4-byte + if (ic + 2 >= nc ) goto unfinished; + c4 = ((const unsigned char *)s)[ic+2]; + if ( 0x80 != (c4 & 0xC0) ) // illegal 4-th byte + wc = -1; + else { // legal 4-byte (produce 2 WCHARs) + ic += 3; + wc = ((c1 & 0x07) << 18) | ((c2 & 0x3F) << 12) | ((c3 & 0x3F) << 6) | (c4 & 0x3F); + wc -= 0x10000; + w1[nw] = (wchar_t)(0xD800 + (wc >> 10)); + if (w2) w2[nw] = w1[nw]; + ++nw; + wc = 0xDC00 + (wc & 0x3FF); +#if 0 + _ASSERTE(nw < wlen); //??? caller should be fixed to avoid this... +#endif + if (nw >= wlen) { --nw; wc = REPLACEMENT; } + } + } + } + } + + if ( wc >= 0 ) { + w1[nw] = (wchar_t)wc; if (w2) w2[nw] = (wchar_t)wc; + } + else { + w1[nw] = REPLACEMENT; if (w2) w2[nw] = L'?'; + } + if (++nw >= wlen) + break; + } + + tail = nc - ic; + return nw; +} + +int Viewer::vread(wchar_t *Buf,int Count, bool Raw, wchar_t *Buf2) +{ if (IsUnicodeCodePage(VM.CodePage)) { // ������ ������, ������ �����! @@ -2765,7 +2859,8 @@ 11110zzz 10zzyyyy 10yyyyxx 10xxxxxx */ - //� ������� ����? ����� ��� ����� �������. +#if 0 + //� ������� ����? ����� ��� ����� �������. while ((c&0xC0) == 0x80) { DWORD Readed=0; @@ -2775,6 +2870,7 @@ ReadSize++; c=*TmpBuf; } +#endif //�������� ������ ���� ��� � ���� ������� ���� ������ ���� �����. DWORD cc=1; @@ -2801,7 +2897,15 @@ } else { - ReadSize = MultiByteToWideChar(VM.CodePage, 0, TmpBuf, ConvertSize, Buf, Count); + if (VM.CodePage != CP_UTF8) + ReadSize = MultiByteToWideChar(VM.CodePage, 0, TmpBuf, ConvertSize, Buf, Count); + else + { + int tail; + ReadSize = (DWORD)utf8_to_WideChar(TmpBuf, ConvertSize, Buf,Buf2, Count, tail); + if (tail) + Reader.Unread(tail); + } } xf_free(TmpBuf); @@ -2812,6 +2916,7 @@ int Viewer::vseek(__int64 Offset,int Whence) { + vgetc_ready = -1; return ViewFile.SetPointer(Offset*(IsUnicodeCodePage(VM.CodePage)?2:1), nullptr, Whence); } @@ -2830,12 +2935,29 @@ bool Viewer::vgetc(WCHAR& C) { - bool Result=false; - - if (vread(&C,1)) - { - Result=true; - } + bool Result; + if (VM.Hex || VM.CodePage != CP_UTF8) + Result = (vread(&C,1) > 0); + else { + if (vgetc_ready >= 0) { + C = (WCHAR)vgetc_ready; vgetc_ready = -1; Result = true; + } + else { + wchar_t w1[4], w2[4]; + int need = 1; + int nw = vread(w1, 4, false, w2); + if (true == (Result = (nw > 0))) { + C = w1[0]; + if (nw > need && 0xD800 == (w1[0] & 0xFC00) && 0xDC00 == (w1[1] & 0xFC00)) { + ++need; vgetc_ready = w1[1]; + } + if (nw > need) { + int tail = GetStrBytesNum(w2+need, nw-need); + Reader.Unread(tail); + } + } + } + } return Result; } |
|
А сдвиг экрана влево - так и должно быть? |
|
Сдвиг при выключеном 'line wrap' |
|
На каком файле это можно увидеть? |
|
FarRus.hlf.m4 ищем "индекс". У меня ширина экрана 100, и уже 2й найденный текст идёт с "<" |
|
Не могу воспроизвести. Точно применён последний патч? Экзэшник с http://sharapov-16.narod.ru/far/index.html даёт этот эффект? По ссылке выше можно скачать пропатченные сборки и изменения far 2.0 (build 1807) и far3. В far3 только последние изменения в hex-mode view, в far2 все изменения viewer-а. |
|
третий патч применен да, дает эффект ищет все нормально, но сдвиг все время присутствует. |
|
|
|
Не вижу этого у себя... (см. аттач) Какая у Вас ось? У меня XP. В принципе конвертация может от этого зависеть (правда на корретных utf-8 врядли). У кого нибудь проявляется этот эффект на XP? |
|
Win7, патч - 3й. Word Wrap должен быть отключён. Приложил screen.png |
|
|
|
Повторил. Спасибо. Буду разбираться... |
|
Нашёл. Чуть позже выложу новый патч. Но боюсь это не последний фикс. Так что проверяйте и сообщайте какие ляпы ещё вылезут... текущий патч: viewer_fixed4.diff |
|
viewer_fixed4.diff (11,462 bytes)
Index: cache.hpp =================================================================== --- cache.hpp (revision 5769) +++ cache.hpp (working copy) @@ -39,6 +39,7 @@ ~CachedRead(); bool Read(LPVOID Data, DWORD DataSize, LPDWORD BytesRead); bool FillBuffer(); + bool Unread(DWORD BytesUnread); void Clear(); private: Index: viewer.hpp =================================================================== --- viewer.hpp (revision 5769) +++ viewer.hpp (working copy) @@ -163,6 +163,9 @@ bool m_bQuickView; UINT DefCodePage; + + int vgetc_ready; + private: virtual void DisplayObject(); @@ -187,7 +190,7 @@ void Search(int Next,int FirstChar); void ConvertToHex(char *SearchStr,int &SearchLength); int HexToNum(int Hex); - int vread(wchar_t *Buf,int Count, bool Raw=false); + int vread(wchar_t *Buf,int Count, bool Raw=false, wchar_t *Buf2=NULL); int vseek(__int64 Offset,int Whence); __int64 vtell(); bool vgetc(WCHAR& C); Index: cache.cpp =================================================================== --- cache.cpp (revision 5769) +++ cache.cpp (working copy) @@ -109,6 +109,18 @@ return Result; } +bool CachedRead::Unread(DWORD BytesUnread) +{ + if (BytesUnread + BytesLeft <= ReadSize) + { + BytesLeft += BytesUnread; + __int64 off = BytesUnread; + file.SetPointer(-off, &LastPtr, FILE_CURRENT); + return true; + } + return false; +} + bool CachedRead::FillBuffer() { bool Result=false; Index: viewer.cpp =================================================================== --- viewer.cpp (revision 5769) +++ viewer.cpp (working copy) @@ -122,6 +122,7 @@ SelectPosOffSet=0; bVE_READ_Sent = false; Signature = false; + vgetc_ready = -1; } @@ -1929,9 +1930,12 @@ if (!ViewFile.Opened()) return; - wchar_t Buf[MAX_VIEWLINE]; + wchar_t Buf[MAX_VIEWLINE], Buf2[ARRAYSIZE(Buf)], *tBuf = Buf, *tBuf2 = NULL; int BufSize,StrPos,Skipped,I,J; + if (!VM.Hex && VM.CodePage == CP_UTF8) + tBuf = tBuf2 = Buf2; + if (FilePos > (__int64)sizeof(Buf)/sizeof(wchar_t)) BufSize=sizeof(Buf)/sizeof(wchar_t); else @@ -1955,7 +1959,7 @@ } vseek(FilePos-(__int64)BufSize,SEEK_SET); - BufSize = vread(Buf,BufSize); + BufSize = vread(Buf, BufSize, false, tBuf2); Skipped=0; if (BufSize>0 && Buf[BufSize-1]==(unsigned int)CRSym) @@ -1979,7 +1983,7 @@ { if (!VM.Wrap) { - FilePos -= GetStrBytesNum(Buf + (I+1), BufSize-(I+1)) + Skipped; + FilePos -= GetStrBytesNum(tBuf + (I+1), BufSize-(I+1)) + Skipped; return; } else @@ -2003,7 +2007,7 @@ if (CalcStrSize(&Buf[J],BufSize-J) <= Width) { - FilePos -= GetStrBytesNum(Buf + J, BufSize-J) + Skipped; + FilePos -= GetStrBytesNum(tBuf + J, BufSize-J) + Skipped; return; } else @@ -2025,7 +2029,7 @@ for (I=Min(Width,BufSize); I>0; I-=5) if (CalcStrSize(&Buf[BufSize-I],I) <= Width) { - FilePos -= GetStrBytesNum(&Buf[BufSize-I], I)+Skipped; + FilePos -= GetStrBytesNum(&tBuf[BufSize-I], I)+Skipped; break; } } @@ -2309,6 +2313,7 @@ ReverseSearch=SearchDlg[SD_CHECKBOX_REVERSE].Selected; SearchRegexp=SearchDlg[SD_CHECKBOX_REGEXP].Selected; + bool t_utf8 = false; if (SearchHex) { strSearchStr = SearchDlg[SD_EDIT_HEX].strData; @@ -2317,6 +2322,7 @@ else { strSearchStr = SearchDlg[SD_EDIT_TEXT].strData; + t_utf8 = VM.CodePage == CP_UTF8; } strLastSearchStr = strSearchStr; @@ -2380,7 +2386,7 @@ if (SearchLength>0 && (!ReverseSearch || LastSelPos>0)) { - wchar_t Buf[8192]; + wchar_t Buf[8192], t_Buf[ARRAYSIZE(Buf)]; __int64 CurPos=LastSelPos; int BufSize=ARRAYSIZE(Buf); @@ -2416,10 +2422,14 @@ vseek(CurPos,SEEK_SET); - if ((ReadSize=vread(Buf,BufSize,SearchHex!=0))<=0) - break; + ReadSize = t_utf8 + ? vread(Buf, BufSize, false, t_Buf) + : vread(Buf, BufSize, SearchHex!=0) + ; + if (ReadSize <= 0) + break; - DWORD CurTime=GetTickCount(); + DWORD CurTime=GetTickCount(); if (CurTime-StartTime>RedrawTimeout) { @@ -2508,29 +2518,38 @@ if (Match) { MatchPos=CurPos+I; + if (t_utf8) + { + MatchPos = CurPos + GetStrBytesNum(t_Buf, I); + SearchLength = GetStrBytesNum(t_Buf+I, SearchLength); + } break; } } - if ((ReverseSearch && CurPos <= 0) || (!ReverseSearch && ReadSize < BufSize)) + if ((ReverseSearch && CurPos <= 0) || (!ReverseSearch && ViewFile.Eof())) break; - if (ReverseSearch) + int asz = ARRAYSIZE(Buf); + if (t_utf8) + asz = GetStrBytesNum(t_Buf, ReadSize); + + if (ReverseSearch) { /* $ 01.08.2000 KM ������� �������� CurPos � ���� Whole words */ if (WholeWords) - CurPos-=ARRAYSIZE(Buf)-SearchLength+1; + CurPos-=asz-SearchLength+1; else - CurPos-=ARRAYSIZE(Buf)-SearchLength; + CurPos-=asz-SearchLength; } else { if (WholeWords) - CurPos+=ARRAYSIZE(Buf)-SearchLength+1; + CurPos+=asz-SearchLength+1; else - CurPos+=ARRAYSIZE(Buf)-SearchLength; + CurPos+=asz-SearchLength; } } } @@ -2705,9 +2724,88 @@ List->MoveData(ViewNamesList); } +#define REPLACEMENT 0xFFFD // L'?' -int Viewer::vread(wchar_t *Buf,int Count, bool Raw) +static int utf8_to_WideChar( + const char *s, int nc, wchar_t *w1,wchar_t *w2, int wlen, int &tail ) { + if ( wlen <= 0 ) + { tail = nc; return 0; } + + int ic = 0, nw = 0, wc; + while ( ic < nc ) + { + unsigned char c4, c3, c2, c1 = ((const unsigned char *)s)[ic++]; + + if (c1 < 0x80) // simple ASCII + wc = (wchar_t)c1; + else if ( c1 < 0xC2 || c1 >= 0xF5 ) // illegal 1-st byte + wc = -1; + else { // multibyte (2, 3, 4) + if (ic + 0 >= nc ) { // unfinished +unfinished: if ( nw > 0 ) tail = nc - ic + 1; + else { tail = 0; w1[0] = REPLACEMENT; if (w2) w2[0] = L'?'; nw = 1; } + return nw; + } + c2 = ((const unsigned char *)s)[ic]; + if ( 0x80 != (c2 & 0xC0) // illegal 2-st byte + || (0xE0 == c1 && c2 <= 0x9F) // illegal 3-byte start (overlaps with 2-byte) + || (0xF0 == c1 && c2 <= 0x8F) // illegal 4-byte start (overlaps with 3-byte) + || (0xF4 == c1 && c2 >= 0x90) // illegal 4-byte (out of unicode range) + ) { + wc = -1; + } + else if ( c1 < 0xE0 ) { // legal 2-byte + ++ic; + wc = ((c1 & 0x1F) << 6) | (c2 & 0x3F); + } + else { // 3 or 4-byte + if (ic + 1 >= nc ) goto unfinished; + c3 = ((const unsigned char *)s)[ic+1]; + if ( 0x80 != (c3 & 0xC0) ) // illegal 3-rd byte + wc = -1; + else if ( c1 < 0xF0 ) { // legal 3-byte + ic += 2; + wc = ((c1 & 0x0F) << 12) | ((c2 & 0x3F) << 6) | (c3 & 0x3F); + } + else { // 4-byte + if (ic + 2 >= nc ) goto unfinished; + c4 = ((const unsigned char *)s)[ic+2]; + if ( 0x80 != (c4 & 0xC0) ) // illegal 4-th byte + wc = -1; + else { // legal 4-byte (produce 2 WCHARs) + ic += 3; + wc = ((c1 & 0x07) << 18) | ((c2 & 0x3F) << 12) | ((c3 & 0x3F) << 6) | (c4 & 0x3F); + wc -= 0x10000; + w1[nw] = (wchar_t)(0xD800 + (wc >> 10)); + if (w2) w2[nw] = w1[nw]; + ++nw; + wc = 0xDC00 + (wc & 0x3FF); +#if 0 + _ASSERTE(nw < wlen); //??? caller should be fixed to avoid this... +#endif + if (nw >= wlen) { --nw; wc = REPLACEMENT; } + } + } + } + } + + if ( wc >= 0 ) { + w1[nw] = (wchar_t)wc; if (w2) w2[nw] = (wchar_t)wc; + } + else { + w1[nw] = REPLACEMENT; if (w2) w2[nw] = L'?'; + } + if (++nw >= wlen) + break; + } + + tail = nc - ic; + return nw; +} + +int Viewer::vread(wchar_t *Buf,int Count, bool Raw, wchar_t *Buf2) +{ if (IsUnicodeCodePage(VM.CodePage)) { // ������ ������, ������ �����! @@ -2765,7 +2863,8 @@ 11110zzz 10zzyyyy 10yyyyxx 10xxxxxx */ - //� ������� ����? ����� ��� ����� �������. +#if 0 + //� ������� ����? ����� ��� ����� �������. while ((c&0xC0) == 0x80) { DWORD Readed=0; @@ -2775,6 +2874,7 @@ ReadSize++; c=*TmpBuf; } +#endif //�������� ������ ���� ��� � ���� ������� ���� ������ ���� �����. DWORD cc=1; @@ -2801,7 +2901,15 @@ } else { - ReadSize = MultiByteToWideChar(VM.CodePage, 0, TmpBuf, ConvertSize, Buf, Count); + if (VM.CodePage != CP_UTF8) + ReadSize = MultiByteToWideChar(VM.CodePage, 0, TmpBuf, ConvertSize, Buf, Count); + else + { + int tail; + ReadSize = (DWORD)utf8_to_WideChar(TmpBuf, ConvertSize, Buf,Buf2, Count, tail); + if (tail) + Reader.Unread(tail); + } } xf_free(TmpBuf); @@ -2812,6 +2920,7 @@ int Viewer::vseek(__int64 Offset,int Whence) { + vgetc_ready = -1; return ViewFile.SetPointer(Offset*(IsUnicodeCodePage(VM.CodePage)?2:1), nullptr, Whence); } @@ -2830,12 +2939,29 @@ bool Viewer::vgetc(WCHAR& C) { - bool Result=false; - - if (vread(&C,1)) - { - Result=true; - } + bool Result; + if (VM.Hex || VM.CodePage != CP_UTF8) + Result = (vread(&C,1) > 0); + else { + if (vgetc_ready >= 0) { + C = (WCHAR)vgetc_ready; vgetc_ready = -1; Result = true; + } + else { + wchar_t w1[4], w2[4]; + int need = 1; + int nw = vread(w1, 4, false, w2); + if (true == (Result = (nw > 0))) { + C = w1[0]; + if (nw > need && 0xD800 == (w1[0] & 0xFC00) && 0xDC00 == (w1[1] & 0xFC00)) { + ++need; vgetc_ready = w1[1]; + } + if (nw > need) { + int tail = GetStrBytesNum(w2+need, nw-need); + Reader.Unread(tail); + } + } + } + } return Result; } @@ -3053,7 +3179,7 @@ if (!ViewFile.Opened()) return; - wchar_t Buf[1024]; + wchar_t Buf[1024], Buf2[ARRAYSIZE(Buf)], *tBuf=(!VM.Hex && VM.CodePage == CP_UTF8) ? Buf2 : NULL; __int64 StartLinePos=-1,SearchLinePos=MatchPos-sizeof(Buf)/sizeof(wchar_t); if (SearchLinePos<0) @@ -3061,12 +3187,15 @@ vseek(SearchLinePos,SEEK_SET); int ReadSize=(int)Min((__int64)ARRAYSIZE(Buf),(__int64)(MatchPos-SearchLinePos)); - ReadSize=vread(Buf,ReadSize); + ReadSize=vread(Buf,ReadSize, false, tBuf); for (int I=ReadSize-1; I>=0; I--) if (Buf[I]==(unsigned int)CRSym) { - StartLinePos=SearchLinePos+I; + if (tBuf) + StartLinePos = MatchPos - GetStrBytesNum(tBuf+I, ReadSize-I); + else + StartLinePos=SearchLinePos+I; break; } |
|
Спасибо, теперь отлично! |
|
build 1903 2useven10: не пропадай :-) |
|
мне казалось последний стабильный 1807. ладно сейчас сделаю 1903... не сделаю. 1903 ещё нет. пропатченный build 1807 (r5497) я выложил. |
|
в 1903 уже vskirdin патч добавил |
|
Понял. Это я с ЛС фар форума сообщение 'смёржил' :) |
Date Modified | Username | Field | Change |
---|---|---|---|
2009-03-13 14:35 | Denis | New Issue | |
2009-03-15 04:59 | Wave | Note Added: 0002784 | |
2009-03-15 07:47 | Denis | Note Added: 0002785 | |
2009-03-15 09:14 | DiRTy_GaRRy | Note Added: 0002786 | |
2009-04-16 09:03 | 8iv | File Added: 2009-04-16_110223.png | |
2009-04-16 09:06 | 8iv | Note Added: 0002955 | |
2009-04-21 09:32 | Denis | Note Added: 0002992 | |
2009-04-21 10:23 | vskirdin | Summary | поиск в UTF-8 файлах не работает => Неверное отображение найденного текста в файлах UTF-8 |
2009-09-04 13:07 | alexy | Relationship added | has duplicate 0000977 |
2009-09-04 13:30 | ajax16384 | Note Added: 0004012 | |
2009-09-04 13:34 | ajax16384 | Note Edited: 0004012 | |
2009-09-08 12:10 | vskirdin | Relationship added | related to 0001047 |
2009-09-08 19:03 | vskirdin | Relationship added | related to 0001046 |
2009-11-03 06:58 | singalen | Note Added: 0004461 | |
2009-11-23 10:56 | alexy | Relationship added | has duplicate 0001173 |
2010-11-09 18:39 | samlyukov | Note Added: 0005950 | |
2010-11-09 18:40 | samlyukov | File Added: !work.diff | |
2010-11-10 11:40 | Volutar | Note Added: 0005952 | |
2010-11-10 11:41 | Volutar | Note Edited: 0005952 | |
2010-11-10 11:42 | Volutar | Note Edited: 0005952 | |
2011-03-07 05:29 | 2useven10 | File Added: viewer.diff | |
2011-03-07 05:33 | 2useven10 | Note Added: 0006471 | |
2011-03-07 07:13 | 2useven10 | File Added: viewer_fixed.diff | |
2011-03-07 07:41 | 2useven10 | File Added: viewer_fixed2.diff | |
2011-03-07 07:53 | vskirdin | Note Added: 0006473 | |
2011-03-07 07:59 | 2useven10 | Note Added: 0006474 | |
2011-03-07 08:27 | DrKnS | Note Added: 0006475 | |
2011-03-07 08:51 | 2useven10 | Note Added: 0006476 | |
2011-03-07 08:57 | 2useven10 | Note Edited: 0006476 | |
2011-03-07 21:39 | alexy | Note Added: 0006479 | |
2011-03-08 08:25 | vskirdin | Note Added: 0006480 | |
2011-03-08 09:02 | vskirdin | Note Added: 0006481 | |
2011-03-08 10:07 | 2useven10 | Note Added: 0006482 | |
2011-03-08 10:17 | 2useven10 | Note Edited: 0006482 | |
2011-03-08 10:24 | 2useven10 | File Added: viewer_fixed3.diff | |
2011-03-08 10:35 | Denis | Note Added: 0006483 | |
2011-03-08 10:37 | Denis | Note Added: 0006484 | |
2011-03-08 10:50 | 2useven10 | Note Added: 0006485 | |
2011-03-08 10:55 | Denis | Note Added: 0006486 | |
2011-03-08 10:58 | Denis | Note Edited: 0006486 | |
2011-03-08 11:03 | 2useven10 | Note Added: 0006487 | |
2011-03-08 11:06 | 2useven10 | Note Edited: 0006487 | |
2011-03-08 11:13 | vskirdin | Note Added: 0006488 | |
2011-03-08 11:14 | vskirdin | Note Edited: 0006488 | |
2011-03-08 11:23 | 2useven10 | File Added: second_match.PNG | |
2011-03-08 11:24 | 2useven10 | Note Added: 0006489 | |
2011-03-08 11:38 | 2useven10 | Note Edited: 0006489 | |
2011-03-08 12:11 | 2useven10 | Note Edited: 0006489 | |
2011-03-08 12:14 | 2useven10 | Note Edited: 0006489 | |
2011-03-08 12:15 | 2useven10 | Note Edited: 0006489 | |
2011-03-08 12:42 | Denis | Note Added: 0006491 | |
2011-03-08 13:22 | Denis | Note Edited: 0006491 | |
2011-03-08 13:23 | Denis | File Added: screen.png | |
2011-03-08 13:26 | Denis | Note Edited: 0006491 | |
2011-03-08 13:27 | Denis | Note Edited: 0006491 | |
2011-03-08 13:55 | 2useven10 | Note Added: 0006492 | |
2011-03-08 18:10 | 2useven10 | Note Added: 0006494 | |
2011-03-08 19:00 | 2useven10 | File Added: viewer_fixed4.diff | |
2011-03-08 19:21 | 2useven10 | Note Edited: 0006494 | |
2011-03-08 19:32 | Denis | Note Added: 0006497 | |
2011-03-09 06:31 | vskirdin | Assigned To | => vskirdin |
2011-03-09 06:31 | vskirdin | Status | new => assigned |
2011-03-09 06:33 | vskirdin | Note Added: 0006498 | |
2011-03-09 06:33 | vskirdin | Status | assigned => feedback |
2011-03-09 07:18 | Denis | Status | feedback => resolved |
2011-03-09 07:25 | 2useven10 | Note Added: 0006499 | |
2011-03-09 07:28 | 2useven10 | Note Edited: 0006499 | |
2011-03-09 07:29 | Denis | Note Added: 0006500 | |
2011-03-09 07:37 | 2useven10 | Note Added: 0006501 | |
2011-03-11 03:32 | 2useven10 | Note Edited: 0006487 | |
2011-03-11 11:31 | 2useven10 | Note Edited: 0006487 | |
2011-03-21 22:11 | 2useven10 | Note Edited: 0006487 | |
2011-03-21 23:55 | alexy | Relationship added | has duplicate 0001707 |
2011-03-21 23:56 | alexy | Build | => 1903 |
2011-03-21 23:56 | alexy | Status | resolved => closed |
2011-03-21 23:56 | alexy | Resolution | open => fixed |
2011-03-21 23:56 | alexy | Fixed in Version | => 3.0 |
2011-03-22 00:12 | alexy | Relationship added | has duplicate 0001562 |
2011-03-22 00:13 | alexy | Relationship deleted | has duplicate 0001562 |
2011-03-22 00:14 | alexy | Relationship added | has duplicate 0001562 |
2011-03-22 00:15 | alexy | Relationship added | has duplicate 0001563 |
2011-03-22 00:22 | alexy | Relationship added | has duplicate 0001216 |
2011-04-11 06:32 | DrKnS | Relationship added | has duplicate 0001749 |
2011-05-01 23:06 | DrKnS | Relationship added | has duplicate 0001761 |
2011-10-07 15:22 | 2useven10 | Relationship added | has duplicate 0001900 |