View Issue Details

IDProjectCategoryView StatusLast Update
0000786Far ManagerViewerpublic2011-03-21 23:56
ReporterDenis Assigned Tovskirdin  
PrioritynormalSeveritymajorReproducibilityalways
Status closedResolutionfixed 
Fixed in Version3.0 
Summary0000786: Неверное отображение найденного текста в файлах UTF-8
DescriptionПоиск в UTF-8 файлах с русским текстом не работает.
Additional InformationFar2 823
TagsNo tags attached.
Build1903

Relationships

has duplicate 0000977 closed Неправильный поиск русского текста в кодировке UTF-8 
has duplicate 0001173 closedalexy Неправильное позицирование курсора при поиске в тексте 
has duplicate 0001707 closedalexy Разный результат поиска при разных начальных позициях 
has duplicate 0001562 closedalexy Viewer does not find string which she must find 
has duplicate 0001563 closed Viewer highlights found string incorrectly 
has duplicate 0001216 closed Search does not work for UNICODE files 
has duplicate 0001749 closed Некорректная работа поиска в юникодных кодовых страницах 
has duplicate 0001761 closed Viewer fails to search in .fb2 file, while Editor does. По F3 в fb2 файлах поиск не работает, а по F4 работает 
has duplicate 0001900 closed2useven10 Search in viewing of UTF-8 file does not find string file Does contain, ANSI and OEM works 
related to 0001047 closed Неправильно находится смещние текста при поиске в просмотре (UTF-8, если есть русский текст) 
related to 0001046 closed Неправильно находится смещние текста при поиске в просмотре (UTF-8, если есть русский текст) 

Activities

Wave

2009-03-15 04:59

reporter   bugnote:0002784

Не работает, если выбраны все кодировки. Когда выбрана UTF-8 — находит.
Far2 823, 825.

Denis

2009-03-15 07:47

reporter   bugnote:0002785

Баг не связан с окном поиска по Alt-F7, а только с Viewer'ом.

DiRTy_GaRRy

2009-03-15 09:14

developer   bugnote:0002786

Wave, заводи новый баг и давай последовательность воспроизведения. У меня по UTF-8 ищет, как при прямом выборе, так и при посике по всем кодировкам.

2009-04-16 08:03

 

2009-04-16_110223.png (35,266 bytes)   
2009-04-16_110223.png (35,266 bytes)   

8iv

2009-04-16 08:06

reporter   bugnote:0002955

870 не работает
1. F3 на FarRus.hlf
2. F7, ввод "plugins", Enter

Находит после 3-4 нажатий пробела и искажает верхние строки (см. приложение)

Denis

2009-04-21 08:32

reporter   bugnote:0002992

Нужно поправить description на "Неверное отображение найденного текста в файлах UTF-8"

ajax16384

2009-09-04 12:30

reporter   bugnote:0004012

Last edited: 2009-09-04 12:34

либо находит лишний раз строку, либо не находит строку вообще.

см мой пример в баге 977

singalen

2009-11-03 06:58

reporter   bugnote:0004461

Отлично проявляется на po-файлах.
build 1184 x86.

samlyukov

2010-11-09 18:39

reporter   bugnote:0005950

каждую неделю постят и постят сабж, может приделать сообщение на F7 - не работает!
например, как в аттаче.

samlyukov

2010-11-09 18:40

reporter  

!work.diff (2,280 bytes)   
Index: viewer.cpp
===================================================================
--- viewer.cpp	(revision 5207)
+++ viewer.cpp	(working copy)
@@ -1356,20 +1356,31 @@
 		}
 		case KEY_F7:
 		{
-			Search(0,0);
+			if (VM.CodePage==CP_UTF8)
+				Message(MSG_WARNING,1,MSG(MViewSearchTitle),MSG(MViewSearchNotSupportUTF8),MSG(MOk));
+			else
+				Search(0,0);
 			return TRUE;
 		}
 		case KEY_SHIFTF7:
 		case KEY_SPACE:
 		{
-			Search(1,0);
+			if (VM.CodePage==CP_UTF8)
+				Message(MSG_WARNING,1,MSG(MViewSearchTitle),MSG(MViewSearchNotSupportUTF8),MSG(MOk));
+			else
+				Search(1,0);
 			return TRUE;
 		}
 		case KEY_ALTF7:
 		{
-			SearchFlags.Set(REVERSE_SEARCH);
-			Search(1,0);
-			SearchFlags.Clear(REVERSE_SEARCH);
+			if (VM.CodePage==CP_UTF8)
+				Message(MSG_WARNING,1,MSG(MViewSearchTitle),MSG(MViewSearchNotSupportUTF8),MSG(MOk));
+			else
+			{
+				SearchFlags.Set(REVERSE_SEARCH);
+				Search(1,0);
+				SearchFlags.Clear(REVERSE_SEARCH);
+			}
 			return TRUE;
 		}
 		case KEY_F8:
Index: farlang.templ.m4
===================================================================
--- farlang.templ.m4	(revision 5207)
+++ farlang.templ.m4	(working copy)
@@ -13992,6 +13992,15 @@
 "Kontynuować wyszukiwanie od końca dokumentu?"
 "Continuar búsqueda desde el final del documento"
 
+MViewSearchNotSupportUTF8
+"Поиск в программе просмотра в кодовой странице UTF-8 не реализован"
+"Поиск в программе просмотра в кодовой странице UTF-8 не реализован"
+"Поиск в программе просмотра в кодовой странице UTF-8 не реализован"
+"Поиск в программе просмотра в кодовой странице UTF-8 не реализован"
+"Поиск в программе просмотра в кодовой странице UTF-8 не реализован"
+"Поиск в программе просмотра в кодовой странице UTF-8 не реализован"
+"Поиск в программе просмотра в кодовой странице UTF-8 не реализован"
+
 MPrintTitle
 l:
 "Печать"
!work.diff (2,280 bytes)   

Volutar

2010-11-10 11:40

reporter   bugnote:0005952

Last edited: 2010-11-10 11:42

Кому и сколько заплатить, чтобы поиск в UTF-8 при просмотре заработал? (это не шутка)

2useven10

2011-03-07 05:29

developer  

viewer.diff (9,730 bytes)   
Index: cache.hpp
===================================================================
--- cache.hpp	(revision 5769)
+++ cache.hpp	(working copy)
@@ -39,6 +39,7 @@
 	~CachedRead();
 	bool Read(LPVOID Data, DWORD DataSize, LPDWORD BytesRead);
 	bool FillBuffer();
+	bool Unread(DWORD BytesUnread);
 	void Clear();
 
 private:
Index: viewer.hpp
===================================================================
--- viewer.hpp	(revision 5769)
+++ viewer.hpp	(working copy)
@@ -163,6 +163,9 @@
 		bool m_bQuickView;
 
 		UINT DefCodePage;
+
+      int vgetc_ready;
+
 	private:
 		virtual void DisplayObject();
 
@@ -187,7 +190,7 @@
 		void Search(int Next,int FirstChar);
 		void ConvertToHex(char *SearchStr,int &SearchLength);
 		int HexToNum(int Hex);
-		int vread(wchar_t *Buf,int Count, bool Raw=false);
+		int vread(wchar_t *Buf,int Count, bool Raw=false, wchar_t *Buf2=NULL);
 		int vseek(__int64 Offset,int Whence);
 		__int64 vtell();
 		bool vgetc(WCHAR& C);
Index: cache.cpp
===================================================================
--- cache.cpp	(revision 5769)
+++ cache.cpp	(working copy)
@@ -109,6 +109,18 @@
 	return Result;
 }
 
+bool CachedRead::Unread(DWORD BytesUnread)
+{
+	if (BytesUnread + BytesLeft <= ReadSize)
+	{
+		BytesLeft += BytesUnread;
+      __int64 off = BytesUnread;
+		file.SetPointer(-off, &LastPtr, FILE_CURRENT);
+		return true;
+	}
+   return false;
+}
+
 bool CachedRead::FillBuffer()
 {
 	bool Result=false;
Index: viewer.cpp
===================================================================
--- viewer.cpp	(revision 5769)
+++ viewer.cpp	(working copy)
@@ -122,6 +122,7 @@
 	SelectPosOffSet=0;
 	bVE_READ_Sent = false;
 	Signature = false;
+   vgetc_ready = -1;
 }
 
 
@@ -1929,9 +1930,12 @@
 	if (!ViewFile.Opened())
 		return;
 
-	wchar_t Buf[MAX_VIEWLINE];
+	wchar_t Buf[MAX_VIEWLINE], Buf2[ARRAYSIZE(Buf)], *tBuf = Buf, *tBuf2 = NULL;
 	int BufSize,StrPos,Skipped,I,J;
 
+   if (!VM.Hex && VM.CodePage == CP_UTF8)
+      tBuf = tBuf2 = Buf2;
+
 	if (FilePos > (__int64)sizeof(Buf)/sizeof(wchar_t))
 		BufSize=sizeof(Buf)/sizeof(wchar_t);
 	else
@@ -1955,7 +1959,7 @@
 	}
 
 	vseek(FilePos-(__int64)BufSize,SEEK_SET);
-	BufSize = vread(Buf,BufSize);
+	BufSize = vread(Buf, BufSize, false, tBuf2);
 	Skipped=0;
 
 	if (BufSize>0 && Buf[BufSize-1]==(unsigned int)CRSym)
@@ -1979,7 +1983,7 @@
 		{
 			if (!VM.Wrap)
 			{
-				FilePos -= GetStrBytesNum(Buf + (I+1), BufSize-(I+1)) + Skipped;
+				FilePos -= GetStrBytesNum(tBuf + (I+1), BufSize-(I+1)) + Skipped;
 				return;
 			}
 			else
@@ -2003,7 +2007,7 @@
 
 						if (CalcStrSize(&Buf[J],BufSize-J) <= Width)
 						{
-							FilePos -= GetStrBytesNum(Buf + J, BufSize-J) + Skipped;
+							FilePos -= GetStrBytesNum(tBuf + J, BufSize-J) + Skipped;
 							return;
 						}
 						else
@@ -2025,7 +2029,7 @@
 	for (I=Min(Width,BufSize); I>0; I-=5)
 		if (CalcStrSize(&Buf[BufSize-I],I) <= Width)
 		{
-			FilePos -= GetStrBytesNum(&Buf[BufSize-I], I)+Skipped;
+			FilePos -= GetStrBytesNum(&tBuf[BufSize-I], I)+Skipped;
 			break;
 		}
 }
@@ -2309,6 +2313,7 @@
 	ReverseSearch=SearchDlg[SD_CHECKBOX_REVERSE].Selected;
 	SearchRegexp=SearchDlg[SD_CHECKBOX_REGEXP].Selected;
 
+	bool t_utf8 = false;
 	if (SearchHex)
 	{
 		strSearchStr = SearchDlg[SD_EDIT_HEX].strData;
@@ -2317,6 +2322,7 @@
 	else
 	{
 		strSearchStr = SearchDlg[SD_EDIT_TEXT].strData;
+		t_utf8 = VM.CodePage == CP_UTF8;
 	}
 
 	strLastSearchStr = strSearchStr;
@@ -2380,7 +2386,7 @@
 
 		if (SearchLength>0 && (!ReverseSearch || LastSelPos>0))
 		{
-			wchar_t Buf[8192];
+			wchar_t Buf[8192], t_Buf[ARRAYSIZE(Buf)];
 			__int64 CurPos=LastSelPos;
 			int BufSize=ARRAYSIZE(Buf);
 
@@ -2416,10 +2422,14 @@
 
 				vseek(CurPos,SEEK_SET);
 
-				if ((ReadSize=vread(Buf,BufSize,SearchHex!=0))<=0)
-					break;
+            ReadSize = t_utf8
+               ? vread(Buf, BufSize, false, t_Buf)
+               : vread(Buf, BufSize, SearchHex!=0)
+            ;
+            if (ReadSize < 0)
+               break;
 
-				DWORD CurTime=GetTickCount();
+            DWORD CurTime=GetTickCount();
 
 				if (CurTime-StartTime>RedrawTimeout)
 				{
@@ -2508,6 +2518,11 @@
 					if (Match)
 					{
 						MatchPos=CurPos+I;
+						if (t_utf8)
+						{
+							MatchPos = CurPos + GetStrBytesNum(t_Buf, I);
+							SearchLength = GetStrBytesNum(t_Buf+I, SearchLength);
+						}
 						break;
 					}
 				}
@@ -2705,9 +2720,88 @@
 		List->MoveData(ViewNamesList);
 }
 
+#define REPLACEMENT 0xFFFD // L'?'
 
-int Viewer::vread(wchar_t *Buf,int Count, bool Raw)
+static int utf8_to_WidwChar(
+   const char *s, int nc, wchar_t *w1,wchar_t *w2, int wlen, int &tail )
 {
+   if ( wlen <= 0 )
+   { tail = nc; return 0; }
+
+   int ic = 0, nw = 0, wc;
+   while ( ic < nc )
+   {
+      unsigned char c4, c3, c2, c1 = ((const unsigned char *)s)[ic++];
+
+      if (c1 < 0x80) // simple ASCII
+         wc = (wchar_t)c1;
+      else if ( c1 < 0xC2 || c1 > 0xF5 ) // illegal 1-st byte
+         wc = -1;
+      else { // multibyte (2, 3, 4)
+         if (ic + 0 >= nc ) { // unfinished
+unfinished: if ( nw > 0 ) tail = nc - ic + 1;
+            else { tail = 0; w1[0] = REPLACEMENT; if (w2) w2[0] = L'?'; nw = 1; }
+            return nw;
+         }
+         c2 = ((const unsigned char *)s)[ic];
+         if ( 0x80 != (c2 & 0xC0)        // illegal 2-st byte
+           || (0xE0 == c1 && c2 <= 0x9F) // illegal 3-byte start (overlaps with 2-byte)
+           || (0xF0 == c1 && c2 <= 0x8F) // illegal 4-byte start (overlaps with 3-byte)
+           || (0xF4 == c1 && c2 >= 0x90) // illegal 4-byte (out of unicode range)
+         ) {
+            wc = -1;
+         }
+         else if ( c1 < 0xE0 ) { // legal 2-byte
+            ++ic;
+            wc = ((c1 & 0x1F) << 6) | (c2 & 0x3F);
+         }
+         else { // 3 or 4-byte
+            if (ic + 1 >= nc ) goto unfinished;
+            c3 = ((const unsigned char *)s)[ic+1];
+            if ( 0x80 != (c3 & 0xC0) ) // illegal 3-rd byte
+               wc = -1;
+            else if ( c1 < 0xF0 ) { // legal 3-byte
+               ic += 2;
+               wc = ((c1 & 0x0F) << 12) | ((c2 & 0x3F) << 6) | (c3 & 0x3F);
+            }
+            else { // 4-byte
+               if (ic + 2 >= nc ) goto unfinished;
+               c4 = ((const unsigned char *)s)[ic+2];
+               if ( 0x80 != (c4 & 0xC0) ) // illegal 4-th byte
+                  wc = -1;
+               else { // legal 4-byte (produce 2 WCHARs)
+                  ic += 3;
+                  wc = ((c1 & 0x07) << 18) | ((c2 & 0x3F) << 12) | ((c3 & 0x3F) << 6) | (c4 & 0x3F);
+                  wc -= 0x10000;
+                  w1[nw] = (wchar_t)(0xD800 + (wc >> 10));
+                  if (w2) w2[nw] = w1[nw];
+                  ++nw;
+                  wc = 0xDC00 + (wc & 0x3FF);
+#if 0
+                  _ASSERTE(nw < wlen); //??? caller should be fixed to avoid this... 
+#endif
+                  if (nw >= wlen) { --nw; wc = REPLACEMENT; }
+               }
+            }
+         }
+      }
+
+      if ( wc >= 0 ) {
+         w1[nw] = (wchar_t)wc; if (w2) w2[nw] = (wchar_t)wc;
+      }
+      else {
+         w1[nw] = REPLACEMENT; if (w2) w2[nw] = L'?';
+      }
+      if (++nw >= wlen)
+         break;
+   }
+
+   tail = nc - ic;
+   return nw;
+}
+
+int Viewer::vread(wchar_t *Buf,int Count, bool Raw, wchar_t *Buf2)
+{
 	if (IsUnicodeCodePage(VM.CodePage))
 	{
 		// ������ ������, ������ �����!
@@ -2765,7 +2859,8 @@
 				11110zzz  10zzyyyy  10yyyyxx  10xxxxxx
 				*/
 
-				//� ������� ����? ����� ��� ����� �������.
+#if 0
+            //� ������� ����? ����� ��� ����� �������.
 				while ((c&0xC0) == 0x80)
 				{
 					DWORD Readed=0;
@@ -2775,6 +2870,7 @@
 					ReadSize++;
 					c=*TmpBuf;
 				}
+#endif
 
 				//�������� ������ ���� ��� � ���� ������� ���� ������ ���� �����.
 				DWORD cc=1;
@@ -2801,7 +2897,15 @@
 		}
 		else
 		{
-			ReadSize = MultiByteToWideChar(VM.CodePage, 0, TmpBuf, ConvertSize, Buf, Count);
+         if (VM.CodePage != CP_UTF8)
+				ReadSize = MultiByteToWideChar(VM.CodePage, 0, TmpBuf, ConvertSize, Buf, Count);
+         else
+         {
+            int tail;
+            ReadSize = (DWORD)utf8_to_WidwChar(TmpBuf, ConvertSize, Buf,Buf2, Count, tail);
+            if (tail)
+					Reader.Unread(tail);
+         }
 		}
 
 		xf_free(TmpBuf);
@@ -2812,6 +2916,7 @@
 
 int Viewer::vseek(__int64 Offset,int Whence)
 {
+   vgetc_ready = -1;
 	return ViewFile.SetPointer(Offset*(IsUnicodeCodePage(VM.CodePage)?2:1), nullptr, Whence);
 }
 
@@ -2830,12 +2935,29 @@
 
 bool Viewer::vgetc(WCHAR& C)
 {
-	bool Result=false;
-
-	if (vread(&C,1))
-	{
-		Result=true;
-	}
+   bool Result;
+   if (VM.Hex || VM.CodePage != CP_UTF8)
+      Result = (vread(&C,1) > 0);
+   else {
+      if (vgetc_ready >= 0) {
+         C = (WCHAR)vgetc_ready; vgetc_ready = -1; Result = true;
+      }
+      else {
+         wchar_t w1[4], w2[4];
+         int need = 1;
+         int nw = vread(w1, 4, false, w2);
+         if (true == (Result = (nw > 0))) {
+            C = w1[0];
+            if (nw > need && 0xD800 == (w1[0] & 0xFC00) && 0xDC00 == (w1[1] & 0xFC00)) {
+               ++need; vgetc_ready = w1[1];
+            }
+            if (nw > need) {
+               int tail = GetStrBytesNum(w2+need, nw-need);
+               Reader.Unread(tail);
+            }
+         }
+      }
+   }
 	return Result;
 }
 
viewer.diff (9,730 bytes)   

2useven10

2011-03-07 05:33

developer   bugnote:0006471

viewer.diff - мои изменения (cache/viewer .cpp/.hpp)
По крайней мере делает вид, что работает...

2useven10

2011-03-07 07:13

developer  

viewer_fixed.diff (9,731 bytes)   
Index: cache.hpp
===================================================================
--- cache.hpp	(revision 5769)
+++ cache.hpp	(working copy)
@@ -39,6 +39,7 @@
 	~CachedRead();
 	bool Read(LPVOID Data, DWORD DataSize, LPDWORD BytesRead);
 	bool FillBuffer();
+	bool Unread(DWORD BytesUnread);
 	void Clear();
 
 private:
Index: viewer.hpp
===================================================================
--- viewer.hpp	(revision 5769)
+++ viewer.hpp	(working copy)
@@ -163,6 +163,9 @@
 		bool m_bQuickView;
 
 		UINT DefCodePage;
+
+      int vgetc_ready;
+
 	private:
 		virtual void DisplayObject();
 
@@ -187,7 +190,7 @@
 		void Search(int Next,int FirstChar);
 		void ConvertToHex(char *SearchStr,int &SearchLength);
 		int HexToNum(int Hex);
-		int vread(wchar_t *Buf,int Count, bool Raw=false);
+		int vread(wchar_t *Buf,int Count, bool Raw=false, wchar_t *Buf2=NULL);
 		int vseek(__int64 Offset,int Whence);
 		__int64 vtell();
 		bool vgetc(WCHAR& C);
Index: cache.cpp
===================================================================
--- cache.cpp	(revision 5769)
+++ cache.cpp	(working copy)
@@ -109,6 +109,18 @@
 	return Result;
 }
 
+bool CachedRead::Unread(DWORD BytesUnread)
+{
+	if (BytesUnread + BytesLeft <= ReadSize)
+	{
+		BytesLeft += BytesUnread;
+      __int64 off = BytesUnread;
+		file.SetPointer(-off, &LastPtr, FILE_CURRENT);
+		return true;
+	}
+   return false;
+}
+
 bool CachedRead::FillBuffer()
 {
 	bool Result=false;
Index: viewer.cpp
===================================================================
--- viewer.cpp	(revision 5769)
+++ viewer.cpp	(working copy)
@@ -122,6 +122,7 @@
 	SelectPosOffSet=0;
 	bVE_READ_Sent = false;
 	Signature = false;
+   vgetc_ready = -1;
 }
 
 
@@ -1929,9 +1930,12 @@
 	if (!ViewFile.Opened())
 		return;
 
-	wchar_t Buf[MAX_VIEWLINE];
+	wchar_t Buf[MAX_VIEWLINE], Buf2[ARRAYSIZE(Buf)], *tBuf = Buf, *tBuf2 = NULL;
 	int BufSize,StrPos,Skipped,I,J;
 
+   if (!VM.Hex && VM.CodePage == CP_UTF8)
+      tBuf = tBuf2 = Buf2;
+
 	if (FilePos > (__int64)sizeof(Buf)/sizeof(wchar_t))
 		BufSize=sizeof(Buf)/sizeof(wchar_t);
 	else
@@ -1955,7 +1959,7 @@
 	}
 
 	vseek(FilePos-(__int64)BufSize,SEEK_SET);
-	BufSize = vread(Buf,BufSize);
+	BufSize = vread(Buf, BufSize, false, tBuf2);
 	Skipped=0;
 
 	if (BufSize>0 && Buf[BufSize-1]==(unsigned int)CRSym)
@@ -1979,7 +1983,7 @@
 		{
 			if (!VM.Wrap)
 			{
-				FilePos -= GetStrBytesNum(Buf + (I+1), BufSize-(I+1)) + Skipped;
+				FilePos -= GetStrBytesNum(tBuf + (I+1), BufSize-(I+1)) + Skipped;
 				return;
 			}
 			else
@@ -2003,7 +2007,7 @@
 
 						if (CalcStrSize(&Buf[J],BufSize-J) <= Width)
 						{
-							FilePos -= GetStrBytesNum(Buf + J, BufSize-J) + Skipped;
+							FilePos -= GetStrBytesNum(tBuf + J, BufSize-J) + Skipped;
 							return;
 						}
 						else
@@ -2025,7 +2029,7 @@
 	for (I=Min(Width,BufSize); I>0; I-=5)
 		if (CalcStrSize(&Buf[BufSize-I],I) <= Width)
 		{
-			FilePos -= GetStrBytesNum(&Buf[BufSize-I], I)+Skipped;
+			FilePos -= GetStrBytesNum(&tBuf[BufSize-I], I)+Skipped;
 			break;
 		}
 }
@@ -2309,6 +2313,7 @@
 	ReverseSearch=SearchDlg[SD_CHECKBOX_REVERSE].Selected;
 	SearchRegexp=SearchDlg[SD_CHECKBOX_REGEXP].Selected;
 
+	bool t_utf8 = false;
 	if (SearchHex)
 	{
 		strSearchStr = SearchDlg[SD_EDIT_HEX].strData;
@@ -2317,6 +2322,7 @@
 	else
 	{
 		strSearchStr = SearchDlg[SD_EDIT_TEXT].strData;
+		t_utf8 = VM.CodePage == CP_UTF8;
 	}
 
 	strLastSearchStr = strSearchStr;
@@ -2380,7 +2386,7 @@
 
 		if (SearchLength>0 && (!ReverseSearch || LastSelPos>0))
 		{
-			wchar_t Buf[8192];
+			wchar_t Buf[8192], t_Buf[ARRAYSIZE(Buf)];
 			__int64 CurPos=LastSelPos;
 			int BufSize=ARRAYSIZE(Buf);
 
@@ -2416,10 +2422,14 @@
 
 				vseek(CurPos,SEEK_SET);
 
-				if ((ReadSize=vread(Buf,BufSize,SearchHex!=0))<=0)
-					break;
+            ReadSize = t_utf8
+               ? vread(Buf, BufSize, false, t_Buf)
+               : vread(Buf, BufSize, SearchHex!=0)
+            ;
+            if (ReadSize <= 0)
+               break;
 
-				DWORD CurTime=GetTickCount();
+            DWORD CurTime=GetTickCount();
 
 				if (CurTime-StartTime>RedrawTimeout)
 				{
@@ -2508,6 +2518,11 @@
 					if (Match)
 					{
 						MatchPos=CurPos+I;
+						if (t_utf8)
+						{
+							MatchPos = CurPos + GetStrBytesNum(t_Buf, I);
+							SearchLength = GetStrBytesNum(t_Buf+I, SearchLength);
+						}
 						break;
 					}
 				}
@@ -2705,9 +2720,88 @@
 		List->MoveData(ViewNamesList);
 }
 
+#define REPLACEMENT 0xFFFD // L'?'
 
-int Viewer::vread(wchar_t *Buf,int Count, bool Raw)
+static int utf8_to_WidwChar(
+   const char *s, int nc, wchar_t *w1,wchar_t *w2, int wlen, int &tail )
 {
+   if ( wlen <= 0 )
+   { tail = nc; return 0; }
+
+   int ic = 0, nw = 0, wc;
+   while ( ic < nc )
+   {
+      unsigned char c4, c3, c2, c1 = ((const unsigned char *)s)[ic++];
+
+      if (c1 < 0x80) // simple ASCII
+         wc = (wchar_t)c1;
+      else if ( c1 < 0xC2 || c1 > 0xF5 ) // illegal 1-st byte
+         wc = -1;
+      else { // multibyte (2, 3, 4)
+         if (ic + 0 >= nc ) { // unfinished
+unfinished: if ( nw > 0 ) tail = nc - ic + 1;
+            else { tail = 0; w1[0] = REPLACEMENT; if (w2) w2[0] = L'?'; nw = 1; }
+            return nw;
+         }
+         c2 = ((const unsigned char *)s)[ic];
+         if ( 0x80 != (c2 & 0xC0)        // illegal 2-st byte
+           || (0xE0 == c1 && c2 <= 0x9F) // illegal 3-byte start (overlaps with 2-byte)
+           || (0xF0 == c1 && c2 <= 0x8F) // illegal 4-byte start (overlaps with 3-byte)
+           || (0xF4 == c1 && c2 >= 0x90) // illegal 4-byte (out of unicode range)
+         ) {
+            wc = -1;
+         }
+         else if ( c1 < 0xE0 ) { // legal 2-byte
+            ++ic;
+            wc = ((c1 & 0x1F) << 6) | (c2 & 0x3F);
+         }
+         else { // 3 or 4-byte
+            if (ic + 1 >= nc ) goto unfinished;
+            c3 = ((const unsigned char *)s)[ic+1];
+            if ( 0x80 != (c3 & 0xC0) ) // illegal 3-rd byte
+               wc = -1;
+            else if ( c1 < 0xF0 ) { // legal 3-byte
+               ic += 2;
+               wc = ((c1 & 0x0F) << 12) | ((c2 & 0x3F) << 6) | (c3 & 0x3F);
+            }
+            else { // 4-byte
+               if (ic + 2 >= nc ) goto unfinished;
+               c4 = ((const unsigned char *)s)[ic+2];
+               if ( 0x80 != (c4 & 0xC0) ) // illegal 4-th byte
+                  wc = -1;
+               else { // legal 4-byte (produce 2 WCHARs)
+                  ic += 3;
+                  wc = ((c1 & 0x07) << 18) | ((c2 & 0x3F) << 12) | ((c3 & 0x3F) << 6) | (c4 & 0x3F);
+                  wc -= 0x10000;
+                  w1[nw] = (wchar_t)(0xD800 + (wc >> 10));
+                  if (w2) w2[nw] = w1[nw];
+                  ++nw;
+                  wc = 0xDC00 + (wc & 0x3FF);
+#if 0
+                  _ASSERTE(nw < wlen); //??? caller should be fixed to avoid this... 
+#endif
+                  if (nw >= wlen) { --nw; wc = REPLACEMENT; }
+               }
+            }
+         }
+      }
+
+      if ( wc >= 0 ) {
+         w1[nw] = (wchar_t)wc; if (w2) w2[nw] = (wchar_t)wc;
+      }
+      else {
+         w1[nw] = REPLACEMENT; if (w2) w2[nw] = L'?';
+      }
+      if (++nw >= wlen)
+         break;
+   }
+
+   tail = nc - ic;
+   return nw;
+}
+
+int Viewer::vread(wchar_t *Buf,int Count, bool Raw, wchar_t *Buf2)
+{
 	if (IsUnicodeCodePage(VM.CodePage))
 	{
 		// ������ ������, ������ �����!
@@ -2765,7 +2859,8 @@
 				11110zzz  10zzyyyy  10yyyyxx  10xxxxxx
 				*/
 
-				//� ������� ����? ����� ��� ����� �������.
+#if 0
+            //� ������� ����? ����� ��� ����� �������.
 				while ((c&0xC0) == 0x80)
 				{
 					DWORD Readed=0;
@@ -2775,6 +2870,7 @@
 					ReadSize++;
 					c=*TmpBuf;
 				}
+#endif
 
 				//�������� ������ ���� ��� � ���� ������� ���� ������ ���� �����.
 				DWORD cc=1;
@@ -2801,7 +2897,15 @@
 		}
 		else
 		{
-			ReadSize = MultiByteToWideChar(VM.CodePage, 0, TmpBuf, ConvertSize, Buf, Count);
+         if (VM.CodePage != CP_UTF8)
+				ReadSize = MultiByteToWideChar(VM.CodePage, 0, TmpBuf, ConvertSize, Buf, Count);
+         else
+         {
+            int tail;
+            ReadSize = (DWORD)utf8_to_WidwChar(TmpBuf, ConvertSize, Buf,Buf2, Count, tail);
+            if (tail)
+					Reader.Unread(tail);
+         }
 		}
 
 		xf_free(TmpBuf);
@@ -2812,6 +2916,7 @@
 
 int Viewer::vseek(__int64 Offset,int Whence)
 {
+   vgetc_ready = -1;
 	return ViewFile.SetPointer(Offset*(IsUnicodeCodePage(VM.CodePage)?2:1), nullptr, Whence);
 }
 
@@ -2830,12 +2935,29 @@
 
 bool Viewer::vgetc(WCHAR& C)
 {
-	bool Result=false;
-
-	if (vread(&C,1))
-	{
-		Result=true;
-	}
+   bool Result;
+   if (VM.Hex || VM.CodePage != CP_UTF8)
+      Result = (vread(&C,1) > 0);
+   else {
+      if (vgetc_ready >= 0) {
+         C = (WCHAR)vgetc_ready; vgetc_ready = -1; Result = true;
+      }
+      else {
+         wchar_t w1[4], w2[4];
+         int need = 1;
+         int nw = vread(w1, 4, false, w2);
+         if (true == (Result = (nw > 0))) {
+            C = w1[0];
+            if (nw > need && 0xD800 == (w1[0] & 0xFC00) && 0xDC00 == (w1[1] & 0xFC00)) {
+               ++need; vgetc_ready = w1[1];
+            }
+            if (nw > need) {
+               int tail = GetStrBytesNum(w2+need, nw-need);
+               Reader.Unread(tail);
+            }
+         }
+      }
+   }
 	return Result;
 }
 
viewer_fixed.diff (9,731 bytes)   

2useven10

2011-03-07 07:41

developer  

viewer_fixed2.diff (9,732 bytes)   
Index: cache.hpp
===================================================================
--- cache.hpp	(revision 5769)
+++ cache.hpp	(working copy)
@@ -39,6 +39,7 @@
 	~CachedRead();
 	bool Read(LPVOID Data, DWORD DataSize, LPDWORD BytesRead);
 	bool FillBuffer();
+	bool Unread(DWORD BytesUnread);
 	void Clear();
 
 private:
Index: viewer.hpp
===================================================================
--- viewer.hpp	(revision 5769)
+++ viewer.hpp	(working copy)
@@ -163,6 +163,9 @@
 		bool m_bQuickView;
 
 		UINT DefCodePage;
+
+      int vgetc_ready;
+
 	private:
 		virtual void DisplayObject();
 
@@ -187,7 +190,7 @@
 		void Search(int Next,int FirstChar);
 		void ConvertToHex(char *SearchStr,int &SearchLength);
 		int HexToNum(int Hex);
-		int vread(wchar_t *Buf,int Count, bool Raw=false);
+		int vread(wchar_t *Buf,int Count, bool Raw=false, wchar_t *Buf2=NULL);
 		int vseek(__int64 Offset,int Whence);
 		__int64 vtell();
 		bool vgetc(WCHAR& C);
Index: cache.cpp
===================================================================
--- cache.cpp	(revision 5769)
+++ cache.cpp	(working copy)
@@ -109,6 +109,18 @@
 	return Result;
 }
 
+bool CachedRead::Unread(DWORD BytesUnread)
+{
+	if (BytesUnread + BytesLeft <= ReadSize)
+	{
+		BytesLeft += BytesUnread;
+      __int64 off = BytesUnread;
+		file.SetPointer(-off, &LastPtr, FILE_CURRENT);
+		return true;
+	}
+   return false;
+}
+
 bool CachedRead::FillBuffer()
 {
 	bool Result=false;
Index: viewer.cpp
===================================================================
--- viewer.cpp	(revision 5769)
+++ viewer.cpp	(working copy)
@@ -122,6 +122,7 @@
 	SelectPosOffSet=0;
 	bVE_READ_Sent = false;
 	Signature = false;
+   vgetc_ready = -1;
 }
 
 
@@ -1929,9 +1930,12 @@
 	if (!ViewFile.Opened())
 		return;
 
-	wchar_t Buf[MAX_VIEWLINE];
+	wchar_t Buf[MAX_VIEWLINE], Buf2[ARRAYSIZE(Buf)], *tBuf = Buf, *tBuf2 = NULL;
 	int BufSize,StrPos,Skipped,I,J;
 
+   if (!VM.Hex && VM.CodePage == CP_UTF8)
+      tBuf = tBuf2 = Buf2;
+
 	if (FilePos > (__int64)sizeof(Buf)/sizeof(wchar_t))
 		BufSize=sizeof(Buf)/sizeof(wchar_t);
 	else
@@ -1955,7 +1959,7 @@
 	}
 
 	vseek(FilePos-(__int64)BufSize,SEEK_SET);
-	BufSize = vread(Buf,BufSize);
+	BufSize = vread(Buf, BufSize, false, tBuf2);
 	Skipped=0;
 
 	if (BufSize>0 && Buf[BufSize-1]==(unsigned int)CRSym)
@@ -1979,7 +1983,7 @@
 		{
 			if (!VM.Wrap)
 			{
-				FilePos -= GetStrBytesNum(Buf + (I+1), BufSize-(I+1)) + Skipped;
+				FilePos -= GetStrBytesNum(tBuf + (I+1), BufSize-(I+1)) + Skipped;
 				return;
 			}
 			else
@@ -2003,7 +2007,7 @@
 
 						if (CalcStrSize(&Buf[J],BufSize-J) <= Width)
 						{
-							FilePos -= GetStrBytesNum(Buf + J, BufSize-J) + Skipped;
+							FilePos -= GetStrBytesNum(tBuf + J, BufSize-J) + Skipped;
 							return;
 						}
 						else
@@ -2025,7 +2029,7 @@
 	for (I=Min(Width,BufSize); I>0; I-=5)
 		if (CalcStrSize(&Buf[BufSize-I],I) <= Width)
 		{
-			FilePos -= GetStrBytesNum(&Buf[BufSize-I], I)+Skipped;
+			FilePos -= GetStrBytesNum(&tBuf[BufSize-I], I)+Skipped;
 			break;
 		}
 }
@@ -2309,6 +2313,7 @@
 	ReverseSearch=SearchDlg[SD_CHECKBOX_REVERSE].Selected;
 	SearchRegexp=SearchDlg[SD_CHECKBOX_REGEXP].Selected;
 
+	bool t_utf8 = false;
 	if (SearchHex)
 	{
 		strSearchStr = SearchDlg[SD_EDIT_HEX].strData;
@@ -2317,6 +2322,7 @@
 	else
 	{
 		strSearchStr = SearchDlg[SD_EDIT_TEXT].strData;
+		t_utf8 = VM.CodePage == CP_UTF8;
 	}
 
 	strLastSearchStr = strSearchStr;
@@ -2380,7 +2386,7 @@
 
 		if (SearchLength>0 && (!ReverseSearch || LastSelPos>0))
 		{
-			wchar_t Buf[8192];
+			wchar_t Buf[8192], t_Buf[ARRAYSIZE(Buf)];
 			__int64 CurPos=LastSelPos;
 			int BufSize=ARRAYSIZE(Buf);
 
@@ -2416,10 +2422,14 @@
 
 				vseek(CurPos,SEEK_SET);
 
-				if ((ReadSize=vread(Buf,BufSize,SearchHex!=0))<=0)
-					break;
+            ReadSize = t_utf8
+               ? vread(Buf, BufSize, false, t_Buf)
+               : vread(Buf, BufSize, SearchHex!=0)
+            ;
+            if (ReadSize <= 0)
+               break;
 
-				DWORD CurTime=GetTickCount();
+            DWORD CurTime=GetTickCount();
 
 				if (CurTime-StartTime>RedrawTimeout)
 				{
@@ -2508,6 +2518,11 @@
 					if (Match)
 					{
 						MatchPos=CurPos+I;
+						if (t_utf8)
+						{
+							MatchPos = CurPos + GetStrBytesNum(t_Buf, I);
+							SearchLength = GetStrBytesNum(t_Buf+I, SearchLength);
+						}
 						break;
 					}
 				}
@@ -2705,9 +2720,88 @@
 		List->MoveData(ViewNamesList);
 }
 
+#define REPLACEMENT 0xFFFD // L'?'
 
-int Viewer::vread(wchar_t *Buf,int Count, bool Raw)
+static int utf8_to_WideChar(
+   const char *s, int nc, wchar_t *w1,wchar_t *w2, int wlen, int &tail )
 {
+   if ( wlen <= 0 )
+   { tail = nc; return 0; }
+
+   int ic = 0, nw = 0, wc;
+   while ( ic < nc )
+   {
+      unsigned char c4, c3, c2, c1 = ((const unsigned char *)s)[ic++];
+
+      if (c1 < 0x80) // simple ASCII
+         wc = (wchar_t)c1;
+      else if ( c1 < 0xC2 || c1 >= 0xF5 ) // illegal 1-st byte
+         wc = -1;
+      else { // multibyte (2, 3, 4)
+         if (ic + 0 >= nc ) { // unfinished
+unfinished: if ( nw > 0 ) tail = nc - ic + 1;
+            else { tail = 0; w1[0] = REPLACEMENT; if (w2) w2[0] = L'?'; nw = 1; }
+            return nw;
+         }
+         c2 = ((const unsigned char *)s)[ic];
+         if ( 0x80 != (c2 & 0xC0)        // illegal 2-st byte
+           || (0xE0 == c1 && c2 <= 0x9F) // illegal 3-byte start (overlaps with 2-byte)
+           || (0xF0 == c1 && c2 <= 0x8F) // illegal 4-byte start (overlaps with 3-byte)
+           || (0xF4 == c1 && c2 >= 0x90) // illegal 4-byte (out of unicode range)
+         ) {
+            wc = -1;
+         }
+         else if ( c1 < 0xE0 ) { // legal 2-byte
+            ++ic;
+            wc = ((c1 & 0x1F) << 6) | (c2 & 0x3F);
+         }
+         else { // 3 or 4-byte
+            if (ic + 1 >= nc ) goto unfinished;
+            c3 = ((const unsigned char *)s)[ic+1];
+            if ( 0x80 != (c3 & 0xC0) ) // illegal 3-rd byte
+               wc = -1;
+            else if ( c1 < 0xF0 ) { // legal 3-byte
+               ic += 2;
+               wc = ((c1 & 0x0F) << 12) | ((c2 & 0x3F) << 6) | (c3 & 0x3F);
+            }
+            else { // 4-byte
+               if (ic + 2 >= nc ) goto unfinished;
+               c4 = ((const unsigned char *)s)[ic+2];
+               if ( 0x80 != (c4 & 0xC0) ) // illegal 4-th byte
+                  wc = -1;
+               else { // legal 4-byte (produce 2 WCHARs)
+                  ic += 3;
+                  wc = ((c1 & 0x07) << 18) | ((c2 & 0x3F) << 12) | ((c3 & 0x3F) << 6) | (c4 & 0x3F);
+                  wc -= 0x10000;
+                  w1[nw] = (wchar_t)(0xD800 + (wc >> 10));
+                  if (w2) w2[nw] = w1[nw];
+                  ++nw;
+                  wc = 0xDC00 + (wc & 0x3FF);
+#if 0
+                  _ASSERTE(nw < wlen); //??? caller should be fixed to avoid this... 
+#endif
+                  if (nw >= wlen) { --nw; wc = REPLACEMENT; }
+               }
+            }
+         }
+      }
+
+      if ( wc >= 0 ) {
+         w1[nw] = (wchar_t)wc; if (w2) w2[nw] = (wchar_t)wc;
+      }
+      else {
+         w1[nw] = REPLACEMENT; if (w2) w2[nw] = L'?';
+      }
+      if (++nw >= wlen)
+         break;
+   }
+
+   tail = nc - ic;
+   return nw;
+}
+
+int Viewer::vread(wchar_t *Buf,int Count, bool Raw, wchar_t *Buf2)
+{
 	if (IsUnicodeCodePage(VM.CodePage))
 	{
 		// ������ ������, ������ �����!
@@ -2765,7 +2859,8 @@
 				11110zzz  10zzyyyy  10yyyyxx  10xxxxxx
 				*/
 
-				//� ������� ����? ����� ��� ����� �������.
+#if 0
+            //� ������� ����? ����� ��� ����� �������.
 				while ((c&0xC0) == 0x80)
 				{
 					DWORD Readed=0;
@@ -2775,6 +2870,7 @@
 					ReadSize++;
 					c=*TmpBuf;
 				}
+#endif
 
 				//�������� ������ ���� ��� � ���� ������� ���� ������ ���� �����.
 				DWORD cc=1;
@@ -2801,7 +2897,15 @@
 		}
 		else
 		{
-			ReadSize = MultiByteToWideChar(VM.CodePage, 0, TmpBuf, ConvertSize, Buf, Count);
+         if (VM.CodePage != CP_UTF8)
+				ReadSize = MultiByteToWideChar(VM.CodePage, 0, TmpBuf, ConvertSize, Buf, Count);
+         else
+         {
+            int tail;
+            ReadSize = (DWORD)utf8_to_WideChar(TmpBuf, ConvertSize, Buf,Buf2, Count, tail);
+            if (tail)
+					Reader.Unread(tail);
+         }
 		}
 
 		xf_free(TmpBuf);
@@ -2812,6 +2916,7 @@
 
 int Viewer::vseek(__int64 Offset,int Whence)
 {
+   vgetc_ready = -1;
 	return ViewFile.SetPointer(Offset*(IsUnicodeCodePage(VM.CodePage)?2:1), nullptr, Whence);
 }
 
@@ -2830,12 +2935,29 @@
 
 bool Viewer::vgetc(WCHAR& C)
 {
-	bool Result=false;
-
-	if (vread(&C,1))
-	{
-		Result=true;
-	}
+   bool Result;
+   if (VM.Hex || VM.CodePage != CP_UTF8)
+      Result = (vread(&C,1) > 0);
+   else {
+      if (vgetc_ready >= 0) {
+         C = (WCHAR)vgetc_ready; vgetc_ready = -1; Result = true;
+      }
+      else {
+         wchar_t w1[4], w2[4];
+         int need = 1;
+         int nw = vread(w1, 4, false, w2);
+         if (true == (Result = (nw > 0))) {
+            C = w1[0];
+            if (nw > need && 0xD800 == (w1[0] & 0xFC00) && 0xDC00 == (w1[1] & 0xFC00)) {
+               ++need; vgetc_ready = w1[1];
+            }
+            if (nw > need) {
+               int tail = GetStrBytesNum(w2+need, nw-need);
+               Reader.Unread(tail);
+            }
+         }
+      }
+   }
 	return Result;
 }
 
viewer_fixed2.diff (9,732 bytes)   

vskirdin

2011-03-07 07:53

administrator   bugnote:0006473

чей код функции utf8_to_WidwChar? откуда взят? какие на него права/лицензии? etc...

2useven10

2011-03-07 07:59

developer   bugnote:0006474

код мой. absolutly free... опечатку в WidwChar я уже исправил.

DrKnS

2011-03-07 08:27

administrator   bugnote:0006475

А чем MultiByteToWideChar(CP_UTF8) не подошёл?

2useven10

2011-03-07 08:51

developer   bugnote:0006476

Last edited: 2011-03-07 08:57

До висты в MultiByteToWideChar() некорректные последовательности просто игнорируются. К тому же у меня делается два варианта перекодировки: один с replacement (0xfffd) для показа, другой с '?' для пересчета смещения (замена должна быть одним байтом в utf-8).

alexy

2011-03-07 21:39

administrator   bugnote:0006479

хм, очень похоже на то что надо. надо закомитить.

vskirdin

2011-03-08 08:25

administrator   bugnote:0006480

рановато коммитить... что-то у меня с поиском не так - разберусь, дам знать

vskirdin

2011-03-08 09:02

administrator   bugnote:0006481

открываю во вьювере FarRus.hlf.m4
Home f7 индекс Enter
блаблабла
в общем только 2 раза находит...

2useven10

2011-03-08 10:07

developer   bugnote:0006482

Last edited: 2011-03-08 10:17

срабатывает условие:
if ((ReverseSearch && CurPos <= 0) || (!ReverseSearch && ReadSize < BufSize))
                    break;

пофиксить думаю будет несложно...
Например так:
if ((ReverseSearch && CurPos <= 0) || (!ReverseSearch && ViewFile.Eof()))
                    break;

2useven10

2011-03-08 10:24

developer  

viewer_fixed3.diff (9,950 bytes)   
Index: cache.hpp
===================================================================
--- cache.hpp	(revision 5769)
+++ cache.hpp	(working copy)
@@ -39,6 +39,7 @@
 	~CachedRead();
 	bool Read(LPVOID Data, DWORD DataSize, LPDWORD BytesRead);
 	bool FillBuffer();
+	bool Unread(DWORD BytesUnread);
 	void Clear();
 
 private:
Index: viewer.hpp
===================================================================
--- viewer.hpp	(revision 5769)
+++ viewer.hpp	(working copy)
@@ -163,6 +163,9 @@
 		bool m_bQuickView;
 
 		UINT DefCodePage;
+
+      int vgetc_ready;
+
 	private:
 		virtual void DisplayObject();
 
@@ -187,7 +190,7 @@
 		void Search(int Next,int FirstChar);
 		void ConvertToHex(char *SearchStr,int &SearchLength);
 		int HexToNum(int Hex);
-		int vread(wchar_t *Buf,int Count, bool Raw=false);
+		int vread(wchar_t *Buf,int Count, bool Raw=false, wchar_t *Buf2=NULL);
 		int vseek(__int64 Offset,int Whence);
 		__int64 vtell();
 		bool vgetc(WCHAR& C);
Index: cache.cpp
===================================================================
--- cache.cpp	(revision 5769)
+++ cache.cpp	(working copy)
@@ -109,6 +109,18 @@
 	return Result;
 }
 
+bool CachedRead::Unread(DWORD BytesUnread)
+{
+	if (BytesUnread + BytesLeft <= ReadSize)
+	{
+		BytesLeft += BytesUnread;
+      __int64 off = BytesUnread;
+		file.SetPointer(-off, &LastPtr, FILE_CURRENT);
+		return true;
+	}
+   return false;
+}
+
 bool CachedRead::FillBuffer()
 {
 	bool Result=false;
Index: viewer.cpp
===================================================================
--- viewer.cpp	(revision 5769)
+++ viewer.cpp	(working copy)
@@ -122,6 +122,7 @@
 	SelectPosOffSet=0;
 	bVE_READ_Sent = false;
 	Signature = false;
+   vgetc_ready = -1;
 }
 
 
@@ -1929,9 +1930,12 @@
 	if (!ViewFile.Opened())
 		return;
 
-	wchar_t Buf[MAX_VIEWLINE];
+	wchar_t Buf[MAX_VIEWLINE], Buf2[ARRAYSIZE(Buf)], *tBuf = Buf, *tBuf2 = NULL;
 	int BufSize,StrPos,Skipped,I,J;
 
+   if (!VM.Hex && VM.CodePage == CP_UTF8)
+      tBuf = tBuf2 = Buf2;
+
 	if (FilePos > (__int64)sizeof(Buf)/sizeof(wchar_t))
 		BufSize=sizeof(Buf)/sizeof(wchar_t);
 	else
@@ -1955,7 +1959,7 @@
 	}
 
 	vseek(FilePos-(__int64)BufSize,SEEK_SET);
-	BufSize = vread(Buf,BufSize);
+	BufSize = vread(Buf, BufSize, false, tBuf2);
 	Skipped=0;
 
 	if (BufSize>0 && Buf[BufSize-1]==(unsigned int)CRSym)
@@ -1979,7 +1983,7 @@
 		{
 			if (!VM.Wrap)
 			{
-				FilePos -= GetStrBytesNum(Buf + (I+1), BufSize-(I+1)) + Skipped;
+				FilePos -= GetStrBytesNum(tBuf + (I+1), BufSize-(I+1)) + Skipped;
 				return;
 			}
 			else
@@ -2003,7 +2007,7 @@
 
 						if (CalcStrSize(&Buf[J],BufSize-J) <= Width)
 						{
-							FilePos -= GetStrBytesNum(Buf + J, BufSize-J) + Skipped;
+							FilePos -= GetStrBytesNum(tBuf + J, BufSize-J) + Skipped;
 							return;
 						}
 						else
@@ -2025,7 +2029,7 @@
 	for (I=Min(Width,BufSize); I>0; I-=5)
 		if (CalcStrSize(&Buf[BufSize-I],I) <= Width)
 		{
-			FilePos -= GetStrBytesNum(&Buf[BufSize-I], I)+Skipped;
+			FilePos -= GetStrBytesNum(&tBuf[BufSize-I], I)+Skipped;
 			break;
 		}
 }
@@ -2309,6 +2313,7 @@
 	ReverseSearch=SearchDlg[SD_CHECKBOX_REVERSE].Selected;
 	SearchRegexp=SearchDlg[SD_CHECKBOX_REGEXP].Selected;
 
+	bool t_utf8 = false;
 	if (SearchHex)
 	{
 		strSearchStr = SearchDlg[SD_EDIT_HEX].strData;
@@ -2317,6 +2322,7 @@
 	else
 	{
 		strSearchStr = SearchDlg[SD_EDIT_TEXT].strData;
+		t_utf8 = VM.CodePage == CP_UTF8;
 	}
 
 	strLastSearchStr = strSearchStr;
@@ -2380,7 +2386,7 @@
 
 		if (SearchLength>0 && (!ReverseSearch || LastSelPos>0))
 		{
-			wchar_t Buf[8192];
+			wchar_t Buf[8192], t_Buf[ARRAYSIZE(Buf)];
 			__int64 CurPos=LastSelPos;
 			int BufSize=ARRAYSIZE(Buf);
 
@@ -2416,10 +2422,14 @@
 
 				vseek(CurPos,SEEK_SET);
 
-				if ((ReadSize=vread(Buf,BufSize,SearchHex!=0))<=0)
-					break;
+            ReadSize = t_utf8
+               ? vread(Buf, BufSize, false, t_Buf)
+               : vread(Buf, BufSize, SearchHex!=0)
+            ;
+            if (ReadSize <= 0)
+               break;
 
-				DWORD CurTime=GetTickCount();
+            DWORD CurTime=GetTickCount();
 
 				if (CurTime-StartTime>RedrawTimeout)
 				{
@@ -2508,11 +2518,16 @@
 					if (Match)
 					{
 						MatchPos=CurPos+I;
+						if (t_utf8)
+						{
+							MatchPos = CurPos + GetStrBytesNum(t_Buf, I);
+							SearchLength = GetStrBytesNum(t_Buf+I, SearchLength);
+						}
 						break;
 					}
 				}
 
-				if ((ReverseSearch && CurPos <= 0) || (!ReverseSearch && ReadSize < BufSize))
+            if ((ReverseSearch && CurPos <= 0) || (!ReverseSearch && ViewFile.Eof()))
 					break;
 
 				if (ReverseSearch)
@@ -2705,9 +2720,88 @@
 		List->MoveData(ViewNamesList);
 }
 
+#define REPLACEMENT 0xFFFD // L'?'
 
-int Viewer::vread(wchar_t *Buf,int Count, bool Raw)
+static int utf8_to_WideChar(
+   const char *s, int nc, wchar_t *w1,wchar_t *w2, int wlen, int &tail )
 {
+   if ( wlen <= 0 )
+   { tail = nc; return 0; }
+
+   int ic = 0, nw = 0, wc;
+   while ( ic < nc )
+   {
+      unsigned char c4, c3, c2, c1 = ((const unsigned char *)s)[ic++];
+
+      if (c1 < 0x80) // simple ASCII
+         wc = (wchar_t)c1;
+      else if ( c1 < 0xC2 || c1 >= 0xF5 ) // illegal 1-st byte
+         wc = -1;
+      else { // multibyte (2, 3, 4)
+         if (ic + 0 >= nc ) { // unfinished
+unfinished: if ( nw > 0 ) tail = nc - ic + 1;
+            else { tail = 0; w1[0] = REPLACEMENT; if (w2) w2[0] = L'?'; nw = 1; }
+            return nw;
+         }
+         c2 = ((const unsigned char *)s)[ic];
+         if ( 0x80 != (c2 & 0xC0)        // illegal 2-st byte
+           || (0xE0 == c1 && c2 <= 0x9F) // illegal 3-byte start (overlaps with 2-byte)
+           || (0xF0 == c1 && c2 <= 0x8F) // illegal 4-byte start (overlaps with 3-byte)
+           || (0xF4 == c1 && c2 >= 0x90) // illegal 4-byte (out of unicode range)
+         ) {
+            wc = -1;
+         }
+         else if ( c1 < 0xE0 ) { // legal 2-byte
+            ++ic;
+            wc = ((c1 & 0x1F) << 6) | (c2 & 0x3F);
+         }
+         else { // 3 or 4-byte
+            if (ic + 1 >= nc ) goto unfinished;
+            c3 = ((const unsigned char *)s)[ic+1];
+            if ( 0x80 != (c3 & 0xC0) ) // illegal 3-rd byte
+               wc = -1;
+            else if ( c1 < 0xF0 ) { // legal 3-byte
+               ic += 2;
+               wc = ((c1 & 0x0F) << 12) | ((c2 & 0x3F) << 6) | (c3 & 0x3F);
+            }
+            else { // 4-byte
+               if (ic + 2 >= nc ) goto unfinished;
+               c4 = ((const unsigned char *)s)[ic+2];
+               if ( 0x80 != (c4 & 0xC0) ) // illegal 4-th byte
+                  wc = -1;
+               else { // legal 4-byte (produce 2 WCHARs)
+                  ic += 3;
+                  wc = ((c1 & 0x07) << 18) | ((c2 & 0x3F) << 12) | ((c3 & 0x3F) << 6) | (c4 & 0x3F);
+                  wc -= 0x10000;
+                  w1[nw] = (wchar_t)(0xD800 + (wc >> 10));
+                  if (w2) w2[nw] = w1[nw];
+                  ++nw;
+                  wc = 0xDC00 + (wc & 0x3FF);
+#if 0
+                  _ASSERTE(nw < wlen); //??? caller should be fixed to avoid this... 
+#endif
+                  if (nw >= wlen) { --nw; wc = REPLACEMENT; }
+               }
+            }
+         }
+      }
+
+      if ( wc >= 0 ) {
+         w1[nw] = (wchar_t)wc; if (w2) w2[nw] = (wchar_t)wc;
+      }
+      else {
+         w1[nw] = REPLACEMENT; if (w2) w2[nw] = L'?';
+      }
+      if (++nw >= wlen)
+         break;
+   }
+
+   tail = nc - ic;
+   return nw;
+}
+
+int Viewer::vread(wchar_t *Buf,int Count, bool Raw, wchar_t *Buf2)
+{
 	if (IsUnicodeCodePage(VM.CodePage))
 	{
 		// ������ ������, ������ �����!
@@ -2765,7 +2859,8 @@
 				11110zzz  10zzyyyy  10yyyyxx  10xxxxxx
 				*/
 
-				//� ������� ����? ����� ��� ����� �������.
+#if 0
+            //� ������� ����? ����� ��� ����� �������.
 				while ((c&0xC0) == 0x80)
 				{
 					DWORD Readed=0;
@@ -2775,6 +2870,7 @@
 					ReadSize++;
 					c=*TmpBuf;
 				}
+#endif
 
 				//�������� ������ ���� ��� � ���� ������� ���� ������ ���� �����.
 				DWORD cc=1;
@@ -2801,7 +2897,15 @@
 		}
 		else
 		{
-			ReadSize = MultiByteToWideChar(VM.CodePage, 0, TmpBuf, ConvertSize, Buf, Count);
+         if (VM.CodePage != CP_UTF8)
+				ReadSize = MultiByteToWideChar(VM.CodePage, 0, TmpBuf, ConvertSize, Buf, Count);
+         else
+         {
+            int tail;
+            ReadSize = (DWORD)utf8_to_WideChar(TmpBuf, ConvertSize, Buf,Buf2, Count, tail);
+            if (tail)
+					Reader.Unread(tail);
+         }
 		}
 
 		xf_free(TmpBuf);
@@ -2812,6 +2916,7 @@
 
 int Viewer::vseek(__int64 Offset,int Whence)
 {
+   vgetc_ready = -1;
 	return ViewFile.SetPointer(Offset*(IsUnicodeCodePage(VM.CodePage)?2:1), nullptr, Whence);
 }
 
@@ -2830,12 +2935,29 @@
 
 bool Viewer::vgetc(WCHAR& C)
 {
-	bool Result=false;
-
-	if (vread(&C,1))
-	{
-		Result=true;
-	}
+   bool Result;
+   if (VM.Hex || VM.CodePage != CP_UTF8)
+      Result = (vread(&C,1) > 0);
+   else {
+      if (vgetc_ready >= 0) {
+         C = (WCHAR)vgetc_ready; vgetc_ready = -1; Result = true;
+      }
+      else {
+         wchar_t w1[4], w2[4];
+         int need = 1;
+         int nw = vread(w1, 4, false, w2);
+         if (true == (Result = (nw > 0))) {
+            C = w1[0];
+            if (nw > need && 0xD800 == (w1[0] & 0xFC00) && 0xDC00 == (w1[1] & 0xFC00)) {
+               ++need; vgetc_ready = w1[1];
+            }
+            if (nw > need) {
+               int tail = GetStrBytesNum(w2+need, nw-need);
+               Reader.Unread(tail);
+            }
+         }
+      }
+   }
 	return Result;
 }
 
viewer_fixed3.diff (9,950 bytes)   

Denis

2011-03-08 10:35

reporter   bugnote:0006483

А сдвиг экрана влево - так и должно быть?

Denis

2011-03-08 10:37

reporter   bugnote:0006484

Сдвиг при выключеном 'line wrap'

2useven10

2011-03-08 10:50

developer   bugnote:0006485

На каком файле это можно увидеть?

Denis

2011-03-08 10:55

reporter   bugnote:0006486

Last edited: 2011-03-08 10:58

FarRus.hlf.m4

ищем "индекс".

У меня ширина экрана 100, и уже 2й найденный текст идёт с "<"

2useven10

2011-03-08 11:03

developer   bugnote:0006487

Last edited: 2011-03-21 22:11

Не могу воспроизвести. Точно применён последний патч?
Экзэшник с http://sharapov-16.narod.ru/far/index.html
даёт этот эффект?

По ссылке выше можно скачать пропатченные сборки и изменения far 2.0 (build 1807)
и far3. В far3 только последние изменения в hex-mode view, в far2 все изменения viewer-а.

vskirdin

2011-03-08 11:13

administrator   bugnote:0006488

Last edited: 2011-03-08 11:14

третий патч применен
да, дает эффект
ищет все нормально, но сдвиг все время присутствует.

2useven10

2011-03-08 11:23

developer  

second_match.PNG (45,368 bytes)   
second_match.PNG (45,368 bytes)   

2useven10

2011-03-08 11:24

developer   bugnote:0006489

Last edited: 2011-03-08 12:15

Не вижу этого у себя... (см. аттач)

Какая у Вас ось? У меня XP. В принципе конвертация может от этого зависеть
(правда на корретных utf-8 врядли). У кого нибудь проявляется этот эффект на XP?

Denis

2011-03-08 12:42

reporter   bugnote:0006491

Last edited: 2011-03-08 13:27

Win7, патч - 3й.

Word Wrap должен быть отключён.
Приложил screen.png

Denis

2011-03-08 13:23

reporter  

screen.png (30,778 bytes)   
screen.png (30,778 bytes)   

2useven10

2011-03-08 13:55

developer   bugnote:0006492

Повторил. Спасибо. Буду разбираться...

2useven10

2011-03-08 18:10

developer   bugnote:0006494

Last edited: 2011-03-08 19:21

Нашёл. Чуть позже выложу новый патч. Но боюсь это не последний фикс.
Так что проверяйте и сообщайте какие ляпы ещё вылезут...

текущий патч: viewer_fixed4.diff

2useven10

2011-03-08 19:00

developer  

viewer_fixed4.diff (11,462 bytes)   
Index: cache.hpp
===================================================================
--- cache.hpp	(revision 5769)
+++ cache.hpp	(working copy)
@@ -39,6 +39,7 @@
 	~CachedRead();
 	bool Read(LPVOID Data, DWORD DataSize, LPDWORD BytesRead);
 	bool FillBuffer();
+	bool Unread(DWORD BytesUnread);
 	void Clear();
 
 private:
Index: viewer.hpp
===================================================================
--- viewer.hpp	(revision 5769)
+++ viewer.hpp	(working copy)
@@ -163,6 +163,9 @@
 		bool m_bQuickView;
 
 		UINT DefCodePage;
+
+      int vgetc_ready;
+
 	private:
 		virtual void DisplayObject();
 
@@ -187,7 +190,7 @@
 		void Search(int Next,int FirstChar);
 		void ConvertToHex(char *SearchStr,int &SearchLength);
 		int HexToNum(int Hex);
-		int vread(wchar_t *Buf,int Count, bool Raw=false);
+		int vread(wchar_t *Buf,int Count, bool Raw=false, wchar_t *Buf2=NULL);
 		int vseek(__int64 Offset,int Whence);
 		__int64 vtell();
 		bool vgetc(WCHAR& C);
Index: cache.cpp
===================================================================
--- cache.cpp	(revision 5769)
+++ cache.cpp	(working copy)
@@ -109,6 +109,18 @@
 	return Result;
 }
 
+bool CachedRead::Unread(DWORD BytesUnread)
+{
+	if (BytesUnread + BytesLeft <= ReadSize)
+	{
+		BytesLeft += BytesUnread;
+      __int64 off = BytesUnread;
+		file.SetPointer(-off, &LastPtr, FILE_CURRENT);
+		return true;
+	}
+   return false;
+}
+
 bool CachedRead::FillBuffer()
 {
 	bool Result=false;
Index: viewer.cpp
===================================================================
--- viewer.cpp	(revision 5769)
+++ viewer.cpp	(working copy)
@@ -122,6 +122,7 @@
 	SelectPosOffSet=0;
 	bVE_READ_Sent = false;
 	Signature = false;
+   vgetc_ready = -1;
 }
 
 
@@ -1929,9 +1930,12 @@
 	if (!ViewFile.Opened())
 		return;
 
-	wchar_t Buf[MAX_VIEWLINE];
+	wchar_t Buf[MAX_VIEWLINE], Buf2[ARRAYSIZE(Buf)], *tBuf = Buf, *tBuf2 = NULL;
 	int BufSize,StrPos,Skipped,I,J;
 
+   if (!VM.Hex && VM.CodePage == CP_UTF8)
+      tBuf = tBuf2 = Buf2;
+
 	if (FilePos > (__int64)sizeof(Buf)/sizeof(wchar_t))
 		BufSize=sizeof(Buf)/sizeof(wchar_t);
 	else
@@ -1955,7 +1959,7 @@
 	}
 
 	vseek(FilePos-(__int64)BufSize,SEEK_SET);
-	BufSize = vread(Buf,BufSize);
+	BufSize = vread(Buf, BufSize, false, tBuf2);
 	Skipped=0;
 
 	if (BufSize>0 && Buf[BufSize-1]==(unsigned int)CRSym)
@@ -1979,7 +1983,7 @@
 		{
 			if (!VM.Wrap)
 			{
-				FilePos -= GetStrBytesNum(Buf + (I+1), BufSize-(I+1)) + Skipped;
+				FilePos -= GetStrBytesNum(tBuf + (I+1), BufSize-(I+1)) + Skipped;
 				return;
 			}
 			else
@@ -2003,7 +2007,7 @@
 
 						if (CalcStrSize(&Buf[J],BufSize-J) <= Width)
 						{
-							FilePos -= GetStrBytesNum(Buf + J, BufSize-J) + Skipped;
+							FilePos -= GetStrBytesNum(tBuf + J, BufSize-J) + Skipped;
 							return;
 						}
 						else
@@ -2025,7 +2029,7 @@
 	for (I=Min(Width,BufSize); I>0; I-=5)
 		if (CalcStrSize(&Buf[BufSize-I],I) <= Width)
 		{
-			FilePos -= GetStrBytesNum(&Buf[BufSize-I], I)+Skipped;
+			FilePos -= GetStrBytesNum(&tBuf[BufSize-I], I)+Skipped;
 			break;
 		}
 }
@@ -2309,6 +2313,7 @@
 	ReverseSearch=SearchDlg[SD_CHECKBOX_REVERSE].Selected;
 	SearchRegexp=SearchDlg[SD_CHECKBOX_REGEXP].Selected;
 
+	bool t_utf8 = false;
 	if (SearchHex)
 	{
 		strSearchStr = SearchDlg[SD_EDIT_HEX].strData;
@@ -2317,6 +2322,7 @@
 	else
 	{
 		strSearchStr = SearchDlg[SD_EDIT_TEXT].strData;
+		t_utf8 = VM.CodePage == CP_UTF8;
 	}
 
 	strLastSearchStr = strSearchStr;
@@ -2380,7 +2386,7 @@
 
 		if (SearchLength>0 && (!ReverseSearch || LastSelPos>0))
 		{
-			wchar_t Buf[8192];
+			wchar_t Buf[8192], t_Buf[ARRAYSIZE(Buf)];
 			__int64 CurPos=LastSelPos;
 			int BufSize=ARRAYSIZE(Buf);
 
@@ -2416,10 +2422,14 @@
 
 				vseek(CurPos,SEEK_SET);
 
-				if ((ReadSize=vread(Buf,BufSize,SearchHex!=0))<=0)
-					break;
+            ReadSize = t_utf8
+               ? vread(Buf, BufSize, false, t_Buf)
+               : vread(Buf, BufSize, SearchHex!=0)
+            ;
+            if (ReadSize <= 0)
+               break;
 
-				DWORD CurTime=GetTickCount();
+            DWORD CurTime=GetTickCount();
 
 				if (CurTime-StartTime>RedrawTimeout)
 				{
@@ -2508,29 +2518,38 @@
 					if (Match)
 					{
 						MatchPos=CurPos+I;
+						if (t_utf8)
+						{
+							MatchPos = CurPos + GetStrBytesNum(t_Buf, I);
+							SearchLength = GetStrBytesNum(t_Buf+I, SearchLength);
+						}
 						break;
 					}
 				}
 
-				if ((ReverseSearch && CurPos <= 0) || (!ReverseSearch && ReadSize < BufSize))
+            if ((ReverseSearch && CurPos <= 0) || (!ReverseSearch && ViewFile.Eof()))
 					break;
 
-				if (ReverseSearch)
+            int asz = ARRAYSIZE(Buf);
+            if (t_utf8)
+               asz = GetStrBytesNum(t_Buf, ReadSize);
+
+            if (ReverseSearch)
 				{
 					/* $ 01.08.2000 KM
 					   ������� �������� CurPos � ���� Whole words
 					*/
 					if (WholeWords)
-						CurPos-=ARRAYSIZE(Buf)-SearchLength+1;
+						CurPos-=asz-SearchLength+1;
 					else
-						CurPos-=ARRAYSIZE(Buf)-SearchLength;
+						CurPos-=asz-SearchLength;
 				}
 				else
 				{
 					if (WholeWords)
-						CurPos+=ARRAYSIZE(Buf)-SearchLength+1;
+						CurPos+=asz-SearchLength+1;
 					else
-						CurPos+=ARRAYSIZE(Buf)-SearchLength;
+						CurPos+=asz-SearchLength;
 				}
 			}
 		}
@@ -2705,9 +2724,88 @@
 		List->MoveData(ViewNamesList);
 }
 
+#define REPLACEMENT 0xFFFD // L'?'
 
-int Viewer::vread(wchar_t *Buf,int Count, bool Raw)
+static int utf8_to_WideChar(
+   const char *s, int nc, wchar_t *w1,wchar_t *w2, int wlen, int &tail )
 {
+   if ( wlen <= 0 )
+   { tail = nc; return 0; }
+
+   int ic = 0, nw = 0, wc;
+   while ( ic < nc )
+   {
+      unsigned char c4, c3, c2, c1 = ((const unsigned char *)s)[ic++];
+
+      if (c1 < 0x80) // simple ASCII
+         wc = (wchar_t)c1;
+      else if ( c1 < 0xC2 || c1 >= 0xF5 ) // illegal 1-st byte
+         wc = -1;
+      else { // multibyte (2, 3, 4)
+         if (ic + 0 >= nc ) { // unfinished
+unfinished: if ( nw > 0 ) tail = nc - ic + 1;
+            else { tail = 0; w1[0] = REPLACEMENT; if (w2) w2[0] = L'?'; nw = 1; }
+            return nw;
+         }
+         c2 = ((const unsigned char *)s)[ic];
+         if ( 0x80 != (c2 & 0xC0)        // illegal 2-st byte
+           || (0xE0 == c1 && c2 <= 0x9F) // illegal 3-byte start (overlaps with 2-byte)
+           || (0xF0 == c1 && c2 <= 0x8F) // illegal 4-byte start (overlaps with 3-byte)
+           || (0xF4 == c1 && c2 >= 0x90) // illegal 4-byte (out of unicode range)
+         ) {
+            wc = -1;
+         }
+         else if ( c1 < 0xE0 ) { // legal 2-byte
+            ++ic;
+            wc = ((c1 & 0x1F) << 6) | (c2 & 0x3F);
+         }
+         else { // 3 or 4-byte
+            if (ic + 1 >= nc ) goto unfinished;
+            c3 = ((const unsigned char *)s)[ic+1];
+            if ( 0x80 != (c3 & 0xC0) ) // illegal 3-rd byte
+               wc = -1;
+            else if ( c1 < 0xF0 ) { // legal 3-byte
+               ic += 2;
+               wc = ((c1 & 0x0F) << 12) | ((c2 & 0x3F) << 6) | (c3 & 0x3F);
+            }
+            else { // 4-byte
+               if (ic + 2 >= nc ) goto unfinished;
+               c4 = ((const unsigned char *)s)[ic+2];
+               if ( 0x80 != (c4 & 0xC0) ) // illegal 4-th byte
+                  wc = -1;
+               else { // legal 4-byte (produce 2 WCHARs)
+                  ic += 3;
+                  wc = ((c1 & 0x07) << 18) | ((c2 & 0x3F) << 12) | ((c3 & 0x3F) << 6) | (c4 & 0x3F);
+                  wc -= 0x10000;
+                  w1[nw] = (wchar_t)(0xD800 + (wc >> 10));
+                  if (w2) w2[nw] = w1[nw];
+                  ++nw;
+                  wc = 0xDC00 + (wc & 0x3FF);
+#if 0
+                  _ASSERTE(nw < wlen); //??? caller should be fixed to avoid this... 
+#endif
+                  if (nw >= wlen) { --nw; wc = REPLACEMENT; }
+               }
+            }
+         }
+      }
+
+      if ( wc >= 0 ) {
+         w1[nw] = (wchar_t)wc; if (w2) w2[nw] = (wchar_t)wc;
+      }
+      else {
+         w1[nw] = REPLACEMENT; if (w2) w2[nw] = L'?';
+      }
+      if (++nw >= wlen)
+         break;
+   }
+
+   tail = nc - ic;
+   return nw;
+}
+
+int Viewer::vread(wchar_t *Buf,int Count, bool Raw, wchar_t *Buf2)
+{
 	if (IsUnicodeCodePage(VM.CodePage))
 	{
 		// ������ ������, ������ �����!
@@ -2765,7 +2863,8 @@
 				11110zzz  10zzyyyy  10yyyyxx  10xxxxxx
 				*/
 
-				//� ������� ����? ����� ��� ����� �������.
+#if 0
+            //� ������� ����? ����� ��� ����� �������.
 				while ((c&0xC0) == 0x80)
 				{
 					DWORD Readed=0;
@@ -2775,6 +2874,7 @@
 					ReadSize++;
 					c=*TmpBuf;
 				}
+#endif
 
 				//�������� ������ ���� ��� � ���� ������� ���� ������ ���� �����.
 				DWORD cc=1;
@@ -2801,7 +2901,15 @@
 		}
 		else
 		{
-			ReadSize = MultiByteToWideChar(VM.CodePage, 0, TmpBuf, ConvertSize, Buf, Count);
+         if (VM.CodePage != CP_UTF8)
+				ReadSize = MultiByteToWideChar(VM.CodePage, 0, TmpBuf, ConvertSize, Buf, Count);
+         else
+         {
+            int tail;
+            ReadSize = (DWORD)utf8_to_WideChar(TmpBuf, ConvertSize, Buf,Buf2, Count, tail);
+            if (tail)
+					Reader.Unread(tail);
+         }
 		}
 
 		xf_free(TmpBuf);
@@ -2812,6 +2920,7 @@
 
 int Viewer::vseek(__int64 Offset,int Whence)
 {
+   vgetc_ready = -1;
 	return ViewFile.SetPointer(Offset*(IsUnicodeCodePage(VM.CodePage)?2:1), nullptr, Whence);
 }
 
@@ -2830,12 +2939,29 @@
 
 bool Viewer::vgetc(WCHAR& C)
 {
-	bool Result=false;
-
-	if (vread(&C,1))
-	{
-		Result=true;
-	}
+   bool Result;
+   if (VM.Hex || VM.CodePage != CP_UTF8)
+      Result = (vread(&C,1) > 0);
+   else {
+      if (vgetc_ready >= 0) {
+         C = (WCHAR)vgetc_ready; vgetc_ready = -1; Result = true;
+      }
+      else {
+         wchar_t w1[4], w2[4];
+         int need = 1;
+         int nw = vread(w1, 4, false, w2);
+         if (true == (Result = (nw > 0))) {
+            C = w1[0];
+            if (nw > need && 0xD800 == (w1[0] & 0xFC00) && 0xDC00 == (w1[1] & 0xFC00)) {
+               ++need; vgetc_ready = w1[1];
+            }
+            if (nw > need) {
+               int tail = GetStrBytesNum(w2+need, nw-need);
+               Reader.Unread(tail);
+            }
+         }
+      }
+   }
 	return Result;
 }
 
@@ -3053,7 +3179,7 @@
 	if (!ViewFile.Opened())
 		return;
 
-	wchar_t Buf[1024];
+	wchar_t Buf[1024], Buf2[ARRAYSIZE(Buf)], *tBuf=(!VM.Hex && VM.CodePage == CP_UTF8) ? Buf2 : NULL;
 	__int64 StartLinePos=-1,SearchLinePos=MatchPos-sizeof(Buf)/sizeof(wchar_t);
 
 	if (SearchLinePos<0)
@@ -3061,12 +3187,15 @@
 
 	vseek(SearchLinePos,SEEK_SET);
 	int ReadSize=(int)Min((__int64)ARRAYSIZE(Buf),(__int64)(MatchPos-SearchLinePos));
-	ReadSize=vread(Buf,ReadSize);
+	ReadSize=vread(Buf,ReadSize, false, tBuf);
 
 	for (int I=ReadSize-1; I>=0; I--)
 		if (Buf[I]==(unsigned int)CRSym)
 		{
-			StartLinePos=SearchLinePos+I;
+         if (tBuf)
+            StartLinePos = MatchPos - GetStrBytesNum(tBuf+I, ReadSize-I);
+         else
+			   StartLinePos=SearchLinePos+I;
 			break;
 		}
 
viewer_fixed4.diff (11,462 bytes)   

Denis

2011-03-08 19:32

reporter   bugnote:0006497

Спасибо, теперь отлично!

vskirdin

2011-03-09 06:33

administrator   bugnote:0006498

build 1903

2useven10: не пропадай :-)

2useven10

2011-03-09 07:25

developer   bugnote:0006499

Last edited: 2011-03-09 07:28

мне казалось последний стабильный 1807.
ладно сейчас сделаю 1903...

не сделаю. 1903 ещё нет.
пропатченный build 1807 (r5497) я выложил.

Denis

2011-03-09 07:29

reporter   bugnote:0006500

в 1903 уже vskirdin патч добавил

2useven10

2011-03-09 07:37

developer   bugnote:0006501

Понял. Это я с ЛС фар форума сообщение 'смёржил' :)

Issue History

Date Modified Username Field Change
2009-03-13 14:35 Denis New Issue
2009-03-15 04:59 Wave Note Added: 0002784
2009-03-15 07:47 Denis Note Added: 0002785
2009-03-15 09:14 DiRTy_GaRRy Note Added: 0002786
2009-04-16 08:03 8iv File Added: 2009-04-16_110223.png
2009-04-16 08:06 8iv Note Added: 0002955
2009-04-21 08:32 Denis Note Added: 0002992
2009-04-21 09:23 vskirdin Summary поиск в UTF-8 файлах не работает => Неверное отображение найденного текста в файлах UTF-8
2009-09-04 12:07 alexy Relationship added has duplicate 0000977
2009-09-04 12:30 ajax16384 Note Added: 0004012
2009-09-04 12:34 ajax16384 Note Edited: 0004012
2009-09-08 11:10 vskirdin Relationship added related to 0001047
2009-09-08 18:03 vskirdin Relationship added related to 0001046
2009-11-03 06:58 singalen Note Added: 0004461
2009-11-23 10:56 alexy Relationship added has duplicate 0001173
2010-11-09 18:39 samlyukov Note Added: 0005950
2010-11-09 18:40 samlyukov File Added: !work.diff
2010-11-10 11:40 Volutar Note Added: 0005952
2010-11-10 11:41 Volutar Note Edited: 0005952
2010-11-10 11:42 Volutar Note Edited: 0005952
2011-03-07 05:29 2useven10 File Added: viewer.diff
2011-03-07 05:33 2useven10 Note Added: 0006471
2011-03-07 07:13 2useven10 File Added: viewer_fixed.diff
2011-03-07 07:41 2useven10 File Added: viewer_fixed2.diff
2011-03-07 07:53 vskirdin Note Added: 0006473
2011-03-07 07:59 2useven10 Note Added: 0006474
2011-03-07 08:27 DrKnS Note Added: 0006475
2011-03-07 08:51 2useven10 Note Added: 0006476
2011-03-07 08:57 2useven10 Note Edited: 0006476
2011-03-07 21:39 alexy Note Added: 0006479
2011-03-08 08:25 vskirdin Note Added: 0006480
2011-03-08 09:02 vskirdin Note Added: 0006481
2011-03-08 10:07 2useven10 Note Added: 0006482
2011-03-08 10:17 2useven10 Note Edited: 0006482
2011-03-08 10:24 2useven10 File Added: viewer_fixed3.diff
2011-03-08 10:35 Denis Note Added: 0006483
2011-03-08 10:37 Denis Note Added: 0006484
2011-03-08 10:50 2useven10 Note Added: 0006485
2011-03-08 10:55 Denis Note Added: 0006486
2011-03-08 10:58 Denis Note Edited: 0006486
2011-03-08 11:03 2useven10 Note Added: 0006487
2011-03-08 11:06 2useven10 Note Edited: 0006487
2011-03-08 11:13 vskirdin Note Added: 0006488
2011-03-08 11:14 vskirdin Note Edited: 0006488
2011-03-08 11:23 2useven10 File Added: second_match.PNG
2011-03-08 11:24 2useven10 Note Added: 0006489
2011-03-08 11:38 2useven10 Note Edited: 0006489
2011-03-08 12:11 2useven10 Note Edited: 0006489
2011-03-08 12:14 2useven10 Note Edited: 0006489
2011-03-08 12:15 2useven10 Note Edited: 0006489
2011-03-08 12:42 Denis Note Added: 0006491
2011-03-08 13:22 Denis Note Edited: 0006491
2011-03-08 13:23 Denis File Added: screen.png
2011-03-08 13:26 Denis Note Edited: 0006491
2011-03-08 13:27 Denis Note Edited: 0006491
2011-03-08 13:55 2useven10 Note Added: 0006492
2011-03-08 18:10 2useven10 Note Added: 0006494
2011-03-08 19:00 2useven10 File Added: viewer_fixed4.diff
2011-03-08 19:21 2useven10 Note Edited: 0006494
2011-03-08 19:32 Denis Note Added: 0006497
2011-03-09 06:31 vskirdin Assigned To => vskirdin
2011-03-09 06:31 vskirdin Status new => assigned
2011-03-09 06:33 vskirdin Note Added: 0006498
2011-03-09 06:33 vskirdin Status assigned => feedback
2011-03-09 07:18 Denis Status feedback => resolved
2011-03-09 07:25 2useven10 Note Added: 0006499
2011-03-09 07:28 2useven10 Note Edited: 0006499
2011-03-09 07:29 Denis Note Added: 0006500
2011-03-09 07:37 2useven10 Note Added: 0006501
2011-03-11 03:32 2useven10 Note Edited: 0006487
2011-03-11 11:31 2useven10 Note Edited: 0006487
2011-03-21 22:11 2useven10 Note Edited: 0006487
2011-03-21 23:55 alexy Relationship added has duplicate 0001707
2011-03-21 23:56 alexy Build => 1903
2011-03-21 23:56 alexy Status resolved => closed
2011-03-21 23:56 alexy Resolution open => fixed
2011-03-21 23:56 alexy Fixed in Version => 3.0
2011-03-22 00:12 alexy Relationship added has duplicate 0001562
2011-03-22 00:13 alexy Relationship deleted has duplicate 0001562
2011-03-22 00:14 alexy Relationship added has duplicate 0001562
2011-03-22 00:15 alexy Relationship added has duplicate 0001563
2011-03-22 00:22 alexy Relationship added has duplicate 0001216
2011-04-11 05:32 DrKnS Relationship added has duplicate 0001749
2011-05-01 22:06 DrKnS Relationship added has duplicate 0001761
2011-10-07 14:22 2useven10 Relationship added has duplicate 0001900