Format won't work after using `% s/old/new/g` in neovim

[scottming]

This is the error

lexical: -32803: %SyntaxError{file: "/Users/scottming/Code/dummy/lib/dummy.ex", line: 9, column: 16, snippet: %{content: "    |>
 Enum.ff(&String.contains?(&1, \"a\"))", offset: 15}, description: "unexpected token: Codepoint failed identifier tokenizatio
n, but a simpler form was found.\n\nGot:\n\n    \"ffï¢\" (code points 0x00066 0x00066 0x000EF 0x000A2)\n\nHint: You could write
the above in a similar way that is accepted by Elixir:\n\n    \"ffïc̸\" (code points 0x00066 0x00066 0x000EF 0x00063 0x00338)\n\n
See https://hexdocs.pm/elixir/unicode-syntax.html for more information."}

defmodule Dummy do
  @moduledoc false

  def hello do
    :world

    ["a", "b", "c"]
    |> Enum.filter(&String.contains?(&1, "x"))
    |> Enum.filter(&String.contains?(&1, "a"))
  end
end

and then % s/filter/f/g, then do format.

I'm not sure if this is a lexical bug or not, will add more information later.

[scottming]

To make this error replicated easier, this is the request of neovim:

first you need to create a file:

defmodule Dummy.Document do
  2
  2
end

then do : % s/2/1g, note, just check the second request, you can find some weird code unit

[DEBUG][2023-05-19 14:58:49] .../vim/lsp/rpc.lua:284	"rpc.send"	{
  jsonrpc = "2.0",
  method = "textDocument/didChange",
  params = {
    contentChanges = { {
        range = {
          ["end"] = {
            character = 3,
            line = 2
          },
          start = {
            character = 2,
            line = 1
          }
        },
        rangeLength = 5,
        text = "1\n  1"
      } },
    textDocument = {
      uri = "file:///Users/scottming/Code/dummy/lib/dummy/document.ex",
      version = 3
    }
  }
}
[DEBUG][2023-05-19 14:58:49] .../vim/lsp/rpc.lua:284	"rpc.send"	{
  jsonrpc = "2.0",
  method = "textDocument/didChange",
  params = {
    contentChanges = { {
        range = {
          ["end"] = {
            character = 3,
            line = 2
          },
          start = {
            character = 2,
            line = 1
          }
        },
        rangeLength = 5,
        text = "1\n  1"
      } },
    textDocument = {
      uri = "file:///Users/scottming/Code/dummy/lib/dummy/document.ex",
      version = 3
    }
  }
}
[DEBUG][2023-05-19 14:58:52] .../vim/lsp/rpc.lua:284	"rpc.send"	{
  jsonrpc = "2.0",
  method = "textDocument/didChange",
  params = {
    contentChanges = { {
        range = {
          ["end"] = {
            character = 5,
            line = 2
          },
          start = {
            character = 2,
            line = 2
          }
        },
        rangeLength = 3,
        text = "2"
      } },
    textDocument = {
      uri = "file:///Users/scottming/Code/dummy/lib/dummy/document.ex",
      version = 4
    }
  }
}
[DEBUG][2023-05-19 14:58:52] .../vim/lsp/rpc.lua:284	"rpc.send"	{
  jsonrpc = "2.0",
  method = "textDocument/didChange",
  params = {
    contentChanges = { {
        range = {
          ["end"] = {
            character = 5,
            line = 2
          },
          start = {
            character = 2,
            line = 2
          }
        },
        rangeLength = 3,
        text = "2"
      } },
    textDocument = {
      uri = "file:///Users/scottming/Code/dummy/lib/dummy/document.ex",
      version = 4
    }
  }
}
[DEBUG][2023-05-19 14:58:53] .../vim/lsp/rpc.lua:284	"rpc.send"	{
  jsonrpc = "2.0",
  method = "textDocument/didChange",
  params = {
    contentChanges = { {
        range = {
          ["end"] = {
            character = 5,
            line = 1
          },
          start = {
            character = 2,
            line = 1
          }
        },
        rangeLength = 3,
        text = "2"
      }, {
        range = {
          ["end"] = {
            character = 3,
            line = 2
          },
          start = {
            character = 2,
            line = 1
          }
        },
        rangeLength = 5,
        text = "1\n  1"
      }, {
        range = {
          ["end"] = {
            character = 5,
            line = 2
          },
          start = {
            character = 2,
            line = 2
          }
        },
        rangeLength = 3,
        text = "2"
      }, {
        range = {
          ["end"] = {
            character = 5,
            line = 1
          },
          start = {
            character = 2,
            line = 1
          }
        },
        rangeLength = 3,
        text = "2"
      }, {
        range = {
          ["end"] = {
            character = 3,
            line = 2
          },
          start = {
            character = 2,
            line = 1
          }
        },
        rangeLength = 5,
        text = "1\n  1"
      } },
    textDocument = {
      uri = "file:///Users/scottming/Code/dummy/lib/dummy/document.ex",
      version = 11
    }
  }
}
[DEBUG][2023-05-19 14:58:53] .../vim/lsp/rpc.lua:284	"rpc.send"	{
  jsonrpc = "2.0",
  method = "textDocument/didChange",
  params = {
    contentChanges = { {
        range = {
          ["end"] = {
            character = 5,
            line = 1
          },
          start = {
            character = 2,
            line = 1
          }
        },
        rangeLength = 3,
        text = "2"
      }, {
        range = {
          ["end"] = {
            character = 3,
            line = 2
          },
          start = {
            character = 2,
            line = 1
          }
        },
        rangeLength = 5,
        text = "1\n  1"
      }, {
        range = {
          ["end"] = {
            character = 5,
            line = 2
          },
          start = {
            character = 2,
            line = 2
          }
        },
        rangeLength = 3,
        text = "2"
      }, {
        range = {
          ["end"] = {
            character = 5,
            line = 1
          },
          start = {
            character = 2,
            line = 1
          }
        },
        rangeLength = 3,
        text = "2"
      }, {
        range = {
          ["end"] = {
            character = 3,
            line = 2
          },
          start = {
            character = 2,
            line = 1
          }
        },
        rangeLength = 5,
        text = "1\n  1"
      } },
    textDocument = {
      uri = "file:///Users/scottming/Code/dummy/lib/dummy/document.ex",
      version = 11
    }
  }
}
[DEBUG][2023-05-19 14:58:59] .../vim/lsp/rpc.lua:284	"rpc.send"	{
  jsonrpc = "2.0",
  method = "textDocument/didChange",
  params = {
    contentChanges = { {
        range = {
          ["end"] = {
            character = 5,
            line = 2
          },
          start = {
            character = 2,
            line = 2
          }
        },
        rangeLength = 3,
        text = "2"
      } },
    textDocument = {
      uri = "file:///Users/scottming/Code/dummy/lib/dummy/document.ex",
      version = 12
    }
  }
}
[DEBUG][2023-05-19 14:58:59] .../vim/lsp/rpc.lua:284	"rpc.send"	{
  jsonrpc = "2.0",
  method = "textDocument/didChange",
  params = {
    contentChanges = { {
        range = {
          ["end"] = {
            character = 5,
            line = 2
          },
          start = {
            character = 2,
            line = 2
          }
        },
        rangeLength = 3,
        text = "2"
      } },
    textDocument = {
      uri = "file:///Users/scottming/Code/dummy/lib/dummy/document.ex",
      version = 12
    }
  }
}
[DEBUG][2023-05-19 14:58:59] .../vim/lsp/rpc.lua:284	"rpc.send"	{
  jsonrpc = "2.0",
  method = "textDocument/didChange",
  params = {
    contentChanges = { {
        range = {
          ["end"] = {
            character = 5,
            line = 1
          },
          start = {
            character = 2,
            line = 1
          }
        },
        rangeLength = 3,
        text = "2"
      }, {
        range = {
          ["end"] = {
            character = 3,
            line = 2
          },
          start = {
            character = 2,
            line = 1
          }
        },
        rangeLength = 5,
        text = "1\n  1"
      } },
    textDocument = {
      uri = "file:///Users/scottming/Code/dummy/lib/dummy/document.ex",
      version = 15
    }
  }
}
[DEBUG][2023-05-19 14:58:59] .../vim/lsp/rpc.lua:284	"rpc.send"	{
  jsonrpc = "2.0",
  method = "textDocument/didChange",
  params = {
    contentChanges = { {
        range = {
          ["end"] = {
            character = 5,
            line = 1
          },
          start = {
            character = 2,
            line = 1
          }
        },
        rangeLength = 3,
        text = "2"
      }, {
        range = {
          ["end"] = {
            character = 3,
            line = 2
          },
          start = {
            character = 2,
            line = 1
          }
        },
        rangeLength = 5,
        text = "1\n  1"
      } },
    textDocument = {
      uri = "file:///Users/scottming/Code/dummy/lib/dummy/document.ex",
      version = 15
    }
  }
}

[scottming]

then I logged the document change event in

  def apply(%__MODULE__{} = state, %DidChange{lsp: event}) do
    uri = event.text_document.uri
    version = event.text_document.version

    b = :erlang.term_to_binary(event)
    version = File.open!("document-change-#{version}.b", [:write])
    IO.binwrite(version, b)

    case Document.Store.get_and_update(
           uri,
           &Document.apply_content_changes(&1, version, event.content_changes)
         ) do
      {:ok, updated_source} ->
        Api.compile_document(state.configuration.project, updated_source)
        {:ok, state}

      error ->
        error
    end
  end

I get these changed event:

iex(1)> for i <- [3, 4, 11, 12, 15], do: File.read!("document-change-#{i}.b") |> :erlang.binary_to_term
[
  %{
    __struct__: Lexical.Protocol.Notifications.DidChange.LSP,
    content_changes: [
      %{
        __struct__: Lexical.Protocol.Types.TextDocument.ContentChangeEvent.TextDocumentContentChangeEvent,
        range: %{
          __struct__: Lexical.Protocol.Types.Range,
          end: %{
            __struct__: Lexical.Protocol.Types.Position,
            character: 3,
            line: 2
          },
          start: %{
            __struct__: Lexical.Protocol.Types.Position,
            character: 2,
            line: 1
          }
        },
        range_length: 5,
        text: "1\n  1"
      }
    ],
    jsonrpc: "2.0",
    method: "textDocument/didChange",
    text_document: %{
      __struct__: Lexical.Protocol.Types.TextDocument.Versioned.Identifier,
      uri: "file:///Users/scottming/Code/dummy/lib/dummy/document.ex",
      version: 3
    }
  },
  %{
    __struct__: Lexical.Protocol.Notifications.DidChange.LSP,
    content_changes: [
      %{
        __struct__: Lexical.Protocol.Types.TextDocument.ContentChangeEvent.TextDocumentContentChangeEvent,
        range: %{
          __struct__: Lexical.Protocol.Types.Range,
          end: %{
            __struct__: Lexical.Protocol.Types.Position,
            character: 5,
            line: 2
          },
          start: %{
            __struct__: Lexical.Protocol.Types.Position,
            character: 2,
            line: 2
          }
        },
        range_length: 3,
        text: "2"
      }
    ],
    jsonrpc: "2.0",
    method: "textDocument/didChange",
    text_document: %{
      __struct__: Lexical.Protocol.Types.TextDocument.Versioned.Identifier,
      uri: "file:///Users/scottming/Code/dummy/lib/dummy/document.ex",
      version: 4
    }
  },
  %{
    __struct__: Lexical.Protocol.Notifications.DidChange.LSP,
    content_changes: [
      %{
        __struct__: Lexical.Protocol.Types.TextDocument.ContentChangeEvent.TextDocumentContentChangeEvent,
        range: %{
          __struct__: Lexical.Protocol.Types.Range,
          end: %{
            __struct__: Lexical.Protocol.Types.Position,
            character: 5,
            line: 1
          },
          start: %{
            __struct__: Lexical.Protocol.Types.Position,
            character: 2,
            line: 1
          }
        },
        range_length: 3,
        text: "2"
      },
      %{
        __struct__: Lexical.Protocol.Types.TextDocument.ContentChangeEvent.TextDocumentContentChangeEvent,
        range: %{
          __struct__: Lexical.Protocol.Types.Range,
          end: %{
            __struct__: Lexical.Protocol.Types.Position,
            character: 3,
            line: 2
          },
          start: %{
            __struct__: Lexical.Protocol.Types.Position,
            character: 2,
            line: 1
          }
        },
        range_length: 5,
        text: "1\n  1"
      },
      %{
        __struct__: Lexical.Protocol.Types.TextDocument.ContentChangeEvent.TextDocumentContentChangeEvent,
        range: %{
          __struct__: Lexical.Protocol.Types.Range,
          end: %{
            __struct__: Lexical.Protocol.Types.Position,
            character: 5,
            line: 2
          },
          start: %{
            __struct__: Lexical.Protocol.Types.Position,
            character: 2,
            line: 2
          }
        },
        range_length: 3,
        text: "2"
      },
      %{
        __struct__: Lexical.Protocol.Types.TextDocument.ContentChangeEvent.TextDocumentContentChangeEvent,
        range: %{
          __struct__: Lexical.Protocol.Types.Range,
          end: %{
            __struct__: Lexical.Protocol.Types.Position,
            character: 5,
            line: 1
          },
          start: %{
            __struct__: Lexical.Protocol.Types.Position,
            character: 2,
            line: 1
          }
        },
        range_length: 3,
        text: "2"
      },
      %{
        __struct__: Lexical.Protocol.Types.TextDocument.ContentChangeEvent.TextDocumentContentChangeEvent,
        range: %{
          __struct__: Lexical.Protocol.Types.Range,
          end: %{
            __struct__: Lexical.Protocol.Types.Position,
            character: 3,
            line: 2
          },
          start: %{
            __struct__: Lexical.Protocol.Types.Position,
            character: 2,
            line: 1
          }
        },
        range_length: 5,
        text: "1\n  1"
      }
    ],
    jsonrpc: "2.0",
    method: "textDocument/didChange",
    text_document: %{
      __struct__: Lexical.Protocol.Types.TextDocument.Versioned.Identifier,
      uri: "file:///Users/scottming/Code/dummy/lib/dummy/document.ex",
      version: 11
    }
  },
  %{
    __struct__: Lexical.Protocol.Notifications.DidChange.LSP,
    content_changes: [
      %{
        __struct__: Lexical.Protocol.Types.TextDocument.ContentChangeEvent.TextDocumentContentChangeEvent,
        range: %{
          __struct__: Lexical.Protocol.Types.Range,
          end: %{
            __struct__: Lexical.Protocol.Types.Position,
            character: 5,
            line: 2
          },
          start: %{
            __struct__: Lexical.Protocol.Types.Position,
            character: 2,
            line: 2
          }
        },
        range_length: 3,
        text: "2"
      }
    ],
    jsonrpc: "2.0",
    method: "textDocument/didChange",
    text_document: %{
      __struct__: Lexical.Protocol.Types.TextDocument.Versioned.Identifier,
      uri: "file:///Users/scottming/Code/dummy/lib/dummy/document.ex",
      version: 12
    }
  },
  %{
    __struct__: Lexical.Protocol.Notifications.DidChange.LSP,
    content_changes: [
      %{
        __struct__: Lexical.Protocol.Types.TextDocument.ContentChangeEvent.TextDocumentContentChangeEvent,
        range: %{
          __struct__: Lexical.Protocol.Types.Range,
          end: %{
            __struct__: Lexical.Protocol.Types.Position,
            character: 5,
            line: 1
          },
          start: %{
            __struct__: Lexical.Protocol.Types.Position,
            character: 2,
            line: 1
          }
        },
        range_length: 3,
        text: "2"
      },
      %{
        __struct__: Lexical.Protocol.Types.TextDocument.ContentChangeEvent.TextDocumentContentChangeEvent,
        range: %{
          __struct__: Lexical.Protocol.Types.Range,
          end: %{
            __struct__: Lexical.Protocol.Types.Position,
            character: 3,
            line: 2
          },
          start: %{
            __struct__: Lexical.Protocol.Types.Position,
            character: 2,
            line: 1
          }
        },
        range_length: 5,
        text: "1\n  1"
      }
    ],
    jsonrpc: "2.0",
    method: "textDocument/didChange",
    text_document: %{
      __struct__: Lexical.Protocol.Types.TextDocument.Versioned.Identifier,
      uri: "file:///Users/scottming/Code/dummy/lib/dummy/document.ex",
      version: 15
    }
  }
]

[scottming]

08:17:39.742 [info] read: "{\"params\":{\"textDocument\":{\"version\":3,\"uri\":\"file:\\/\\/\\/Users\\/scottming\\/Code\\/dummy\\/lib\\/dummy\\/document.ex\"},\"contentChanges\":[{\"range\":{\"end\":{\"line\":2,\"character\":3},\"start\":{\"line\":1,\"character\":2}},\"text\":\"g\\n  g\",\"rangeLength\":5}]},\"jsonrpc\":\"2.0\",\"method\":\"textDocument\\/didChange\"}"
08:17:40.078 [info] read: "{\"params\":{\"textDocument\":{\"version\":4,\"uri\":\"file:\\/\\/\\/Users\\/scottming\\/Code\\/dummy\\/lib\\/dummy\\/document.ex\"},\"contentChanges\":[{\"range\":{\"end\":{\"line\":2,\"character\":5},\"start\":{\"line\":2,\"character\":2}},\"text\":\"2\",\"rangeLength\":3}]},\"jsonrpc\":\"2.0\",\"method\":\"textDocument\\/didChange\"}"
08:17:40.228 [info] read: "{\"params\":{\"textDocument\":{\"version\":5,\"uri\":\"file:\\/\\/\\/Users\\/scottming\\/Code\\/dummy\\/lib\\/dummy\\/document.ex\"},\"contentChanges\":[{\"range\":{\"end\":{\"line\":1,\"character\":5},\"start\":{\"line\":1,\"character\":2}},\"text\":\"2\",\"rangeLength\":3}]},\"jsonrpc\":\"2.0\",\"method\":\"textDocument\\/didChange\"}"
08:17:40.376 [info] read: "{\"params\":{\"textDocument\":{\"version\":7,\"uri\":\"file:\\/\\/\\/Users\\/scottming\\/Code\\/dummy\\/lib\\/dummy\\/document.ex\"},\"contentChanges\":[{\"range\":{\"end\":{\"line\":2,\"character\":3},\"start\":{\"line\":1,\"character\":2}},\"text\":\"1\\n  1\",\"rangeLength\":5}]},\"jsonrpc\":\"2.0\",\"method\":\"textDocument\\/didChange\"}"
08:17:43.247 [info] read: "{\"params\":{\"textDocument\":{\"version\":8,\"uri\":\"file:\\/\\/\\/Users\\/scottming\\/Code\\/dummy\\/lib\\/dummy\\/document.ex\"},\"contentChanges\":[{\"range\":{\"end\":{\"line\":2,\"character\":5},\"start\":{\"line\":2,\"character\":2}},\"text\":\"2\",\"rangeLength\":3}]},\"jsonrpc\":\"2.0\",\"method\":\"textDocument\\/didChange\"}"
08:17:43.396 [info] read: "{\"params\":{\"textDocument\":{\"version\":11,\"uri\":\"file:\\/\\/\\/Users\\/scottming\\/Code\\/dummy\\/lib\\/dummy\\/document.ex\"},\"contentChanges\":[{\"range\":{\"end\":{\"line\":1,\"character\":5},\"start\":{\"line\":1,\"character\":2}},\"text\":\"2\",\"rangeLength\":3},{\"range\":{\"end\":{\"line\":2,\"character\":3},\"start\":{\"line\":1,\"character\":2}},\"text\":\"1\\n  1\",\"rangeLength\":5}]},\"jsonrpc\":\"2.0\",\"method\":\"textDocument\\/didChange\"}"
08:17:44.782 [info] read: "{\"params\":{\"textDocument\":{\"uri\":\"file:\\/\\/\\/Users\\/scottming\\/Code\\/dummy\\/lib\\/dummy\\/document.ex\"},\"options\":{\"insertSpaces\":true,\"tabSize\":2}},\"method\":\"textDocument\\/formatting\",\"jsonrpc\":\"2.0\",\"id\":2}"
08:17:44.790 [error] Formatter failed %SyntaxError{file: "/Users/scottming/Code/dummy/lib/dummy/document.ex", line: 3, column: 6, snippet: %{content: "  11g", offset: 5}, description: "unexpected token: Codepoint failed identifier tokenization, but a simpler form was found.\n\nGot:\n\n    \"ï¢\" (code points 0x000EF 0x000A2)\n\nHint: You could write the above in a similar way that is accepted by Elixir:\n\n    \"ïc̸\" (code points 0x000EF 0x00063 0x00338)\n\nSee https://hexdocs.pm/elixir/unicode-syntax.html for more information."}

[scohen]

We looked into this and here are my conclusions:

1. You have a file that is totally in ascii text
2. You make a global search and replace of 1 with 2. Both are ascii
3. Neovim reports completely a completely bonkers text edit to us, with the text being totally ascii (though the ["end"] key is nutso...)
4. Neovim doesn't set any content-type
5. Lexical receives a text edit with the "text" field being non-ascii text

Lexical then treats this non-ascii text correctly and applies the edit successfully, though the formatter fails. I don't think this is a lexical bug because we never received a content-type telling us this isn't a utf-8 request, and we never modify the contents of any of the JSON-RPC fields.

I've tried to replicate in both vscode and emacs, and both can do global search and replace successfully.

My conclusion is that this isn't a lexical bug.