Reading a file line by line in Go

https://stackoverflow.com/questions/8757389/reading-a-file-line-by-line-in-go

I’m unable to find file.ReadLine function in Go. I can figure out how to quickly write one, but I am just wondering if I’m overlooking something here. How does one read a file line by line?


As of Go1.1, bufio.Scanner is the best way to do this. - Malcolm


NOTE: The accepted answer was correct in early versions of Go. See the highest voted answer contains the more recent idiomatic way to achieve this.

There is function ReadLine in package bufio.

Please note that if the line does not fit into the read buffer, the function will return an incomplete line. If you want to always read a whole line in your program by a single call to a function, you will need to encapsulate the ReadLine function into your own function which calls ReadLine in a for-loop.

bufio.ReadString('\n') isn’t fully equivalent to ReadLine because ReadString is unable to handle the case when the last line of a file does not end with the newline character.


From the docs: “ReadLine is a low-level line-reading primitive. Most callers should use ReadBytes(‘\n’) or ReadString(‘\n’) instead or use a Scanner.” - mdwhatcott
@mdwhatcott why does it matter that its a “ low-level line-reading primitive”? How does that reach to the conclusion that “Most callers should use ReadBytes(‘\n’) or ReadString(‘\n’) instead or use a Scanner.”? - Charlie Parker
@CharlieParker - Not sure, just quoting the docs to add context. - mdwhatcott
From the same docs.. “If ReadString encounters an error before finding a delimiter, it returns the data read before the error and the error itself (often io.EOF).” So you can just check for io.EOF error and know your are done. - eduncan911
Note that a read or write can fail due to an interrupted system call, which results in less than the expected number of bytes being read or written. - Justin Swanhart


Use:

  • reader.ReadString('\n')
    If you don’t mind that the line could be very long (i.e. use a lot of RAM). It keeps the \n at the end of the string returned.
  • reader.ReadLine()
    If you care about limiting RAM consumption and don’t mind the extra work of handling the case where the line is greater than the reader’s buffer size.

I tested the various solutions suggested by writing a program to test the scenarios which are identified as problems in other answers:

  • A file with a 4MB line.
  • A file which doesn’t end with a line break.

I found that:

  • The Scanner solution does not handle long lines.
  • The ReadLine solution is complex to implement.
  • The ReadString solution is the simplest and works for long lines.

Here is code which demonstrates each solution, it can be run via go run main.go, or at https://play.golang.org/p/RAW3sGblbas

  1. package main
  2. import (
  3. "bufio"
  4. "bytes"
  5. "fmt"
  6. "io"
  7. "os"
  8. )
  9. func readFileWithReadString(fn string) (err error) {
  10. fmt.Println("readFileWithReadString")
  11. file, err := os.Open(fn)
  12. if err != nil {
  13. return err
  14. }
  15. defer file.Close()
  16. // Start reading from the file with a reader.
  17. reader := bufio.NewReader(file)
  18. var line string
  19. for {
  20. line, err = reader.ReadString('\n')
  21. if err != nil && err != io.EOF {
  22. break
  23. }
  24. // Process the line here.
  25. fmt.Printf(" > Read %d characters\n", len(line))
  26. fmt.Printf(" > > %s\n", limitLength(line, 50))
  27. if err != nil {
  28. break
  29. }
  30. }
  31. if err != io.EOF {
  32. fmt.Printf(" > Failed with error: %v\n", err)
  33. return err
  34. }
  35. return
  36. }
  37. func readFileWithScanner(fn string) (err error) {
  38. fmt.Println("readFileWithScanner (scanner fails with long lines)")
  39. // Don't use this, it doesn't work with long lines...
  40. file, err := os.Open(fn)
  41. if err != nil {
  42. return err
  43. }
  44. defer file.Close()
  45. // Start reading from the file using a scanner.
  46. scanner := bufio.NewScanner(file)
  47. for scanner.Scan() {
  48. line := scanner.Text()
  49. // Process the line here.
  50. fmt.Printf(" > Read %d characters\n", len(line))
  51. fmt.Printf(" > > %s\n", limitLength(line, 50))
  52. }
  53. if scanner.Err() != nil {
  54. fmt.Printf(" > Failed with error %v\n", scanner.Err())
  55. return scanner.Err()
  56. }
  57. return
  58. }
  59. func readFileWithReadLine(fn string) (err error) {
  60. fmt.Println("readFileWithReadLine")
  61. file, err := os.Open(fn)
  62. if err != nil {
  63. return err
  64. }
  65. defer file.Close()
  66. // Start reading from the file with a reader.
  67. reader := bufio.NewReader(file)
  68. for {
  69. var buffer bytes.Buffer
  70. var l []byte
  71. var isPrefix bool
  72. for {
  73. l, isPrefix, err = reader.ReadLine()
  74. buffer.Write(l)
  75. // If we've reached the end of the line, stop reading.
  76. if !isPrefix {
  77. break
  78. }
  79. // If we're at the EOF, break.
  80. if err != nil {
  81. if err != io.EOF {
  82. return err
  83. }
  84. break
  85. }
  86. }
  87. line := buffer.String()
  88. // Process the line here.
  89. fmt.Printf(" > Read %d characters\n", len(line))
  90. fmt.Printf(" > > %s\n", limitLength(line, 50))
  91. if err == io.EOF {
  92. break
  93. }
  94. }
  95. if err != io.EOF {
  96. fmt.Printf(" > Failed with error: %v\n", err)
  97. return err
  98. }
  99. return
  100. }
  101. func main() {
  102. testLongLines()
  103. testLinesThatDoNotFinishWithALinebreak()
  104. }
  105. func testLongLines() {
  106. fmt.Println("Long lines")
  107. fmt.Println()
  108. createFileWithLongLine("longline.txt")
  109. readFileWithReadString("longline.txt")
  110. fmt.Println()
  111. readFileWithScanner("longline.txt")
  112. fmt.Println()
  113. readFileWithReadLine("longline.txt")
  114. fmt.Println()
  115. }
  116. func testLinesThatDoNotFinishWithALinebreak() {
  117. fmt.Println("No linebreak")
  118. fmt.Println()
  119. createFileThatDoesNotEndWithALineBreak("nolinebreak.txt")
  120. readFileWithReadString("nolinebreak.txt")
  121. fmt.Println()
  122. readFileWithScanner("nolinebreak.txt")
  123. fmt.Println()
  124. readFileWithReadLine("nolinebreak.txt")
  125. fmt.Println()
  126. }
  127. func createFileThatDoesNotEndWithALineBreak(fn string) (err error) {
  128. file, err := os.Create(fn)
  129. if err != nil {
  130. return err
  131. }
  132. defer file.Close()
  133. w := bufio.NewWriter(file)
  134. w.WriteString("Does not end with linebreak.")
  135. w.Flush()
  136. return
  137. }
  138. func createFileWithLongLine(fn string) (err error) {
  139. file, err := os.Create(fn)
  140. if err != nil {
  141. return err
  142. }
  143. defer file.Close()
  144. w := bufio.NewWriter(file)
  145. fs := 1024 * 1024 * 4 // 4MB
  146. // Create a 4MB long line consisting of the letter a.
  147. for i := 0; i < fs; i++ {
  148. w.WriteRune('a')
  149. }
  150. // Terminate the line with a break.
  151. w.WriteRune('\n')
  152. // Put in a second line, which doesn't have a linebreak.
  153. w.WriteString("Second line.")
  154. w.Flush()
  155. return
  156. }
  157. func limitLength(s string, length int) string {
  158. if len(s) < length {
  159. return s
  160. }
  161. return s[:length]
  162. }

I tested on:

  • go version go1.15 darwin/amd64
  • go version go1.7 windows/amd64
  • go version go1.6.3 linux/amd64
  • go version go1.7.4 darwin/amd64

The test program outputs:

  1. Long lines
  2. readFileWithReadString
  3. > Read 4194305 characters
  4. > > aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
  5. > Read 12 characters
  6. > > Second line.
  7. readFileWithScanner (scanner fails with long lines)
  8. > Failed with error bufio.Scanner: token too long
  9. readFileWithReadLine
  10. > Read 4194304 characters
  11. > > aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
  12. > Read 12 characters
  13. > > Second line.
  14. > Read 0 characters
  15. > >
  16. No linebreak
  17. readFileWithReadString
  18. > Read 28 characters
  19. > > Does not end with linebreak.
  20. readFileWithScanner (scanner fails with long lines)
  21. > Read 28 characters
  22. > > Does not end with linebreak.
  23. readFileWithReadLine
  24. > Read 28 characters
  25. > > Does not end with linebreak.
  26. > Read 0 characters
  27. > >

The defer file.Close() should be after the error check; otherwise on error it will panic. - mlg
Scanner solution does handle the long lines if you configure it like so. See: golang.org/pkg/bufio/#Scanner.Buffer - Inanc Gumus
You should check the error properly as seen in the docs: play.golang.org/p/5CCPzVTSj6 i.e. if err == io.EOF {break} else {return err} - Chuque


Another method is to use the io/ioutil and strings libraries to read the entire file’s bytes, convert them into a string and split them using a “\n“ (newline) character as the delimiter, for example:

  1. import (
  2. "io/ioutil"
  3. "strings"
  4. )
  5. func main() {
  6. bytesRead, _ := ioutil.ReadFile("something.txt")
  7. file_content := string(bytesRead)
  8. lines := strings.Split(file_content, "\n")
  9. }

Technically you’re not reading the file line-by-line, however you are able to parse each line using this technique. This method is applicable to smaller files. If you’re attempting to parse a massive file use one of the techniques that reads line-by-line.


Example from this gist

  1. func readLine(path string) {
  2. inFile, err := os.Open(path)
  3. if err != nil {
  4. fmt.Println(err.Error() + `: ` + path)
  5. return
  6. }
  7. defer inFile.Close()
  8. scanner := bufio.NewScanner(inFile)
  9. for scanner.Scan() {
  10. fmt.Println(scanner.Text()) // the line
  11. }
  12. }

but this gives an error when there is a line that larger than Scanner’s buffer.

When that happened, what I do is use reader := bufio.NewReader(inFile) create and concat my own buffer either using ch, err := reader.ReadByte() or len, err := reader.Read(myBuffer)

Another way that I use (replace os.Stdin with file like above), this one concats when lines are long (isPrefix) and ignores empty lines:

  1. func readLines() []string {
  2. r := bufio.NewReader(os.Stdin)
  3. bytes := []byte{}
  4. lines := []string{}
  5. for {
  6. line, isPrefix, err := r.ReadLine()
  7. if err != nil {
  8. break
  9. }
  10. bytes = append(bytes, line...)
  11. if !isPrefix {
  12. str := strings.TrimSpace(string(bytes))
  13. if len(str) > 0 {
  14. lines = append(lines, str)
  15. bytes = []byte{}
  16. }
  17. }
  18. }
  19. if len(bytes) > 0 {
  20. lines = append(lines, string(bytes))
  21. }
  22. return lines
  23. }

care to explain why -1? - Kokizzu
I think it;s a little bit overcomplicated this solution, don’t you ? - Decebal


There two common way to read file line by line.

  1. Use bufio.Scanner
  2. Use ReadString/ReadBytes/… in bufio.Reader

In my testcase, ~250MB, ~2,500,000 lines, bufio.Scanner(time used: 0.395491384s) is faster than bufio.Reader.ReadString(time_used: 0.446867622s).

Source code: https://github.com/xpzouying/go-practice/tree/master/read_file_line_by_line

Read file use bufio.Scanner,

  1. func scanFile() {
  2. f, err := os.OpenFile(logfile, os.O_RDONLY, os.ModePerm)
  3. if err != nil {
  4. log.Fatalf("open file error: %v", err)
  5. return
  6. }
  7. defer f.Close()
  8. sc := bufio.NewScanner(f)
  9. for sc.Scan() {
  10. _ = sc.Text() // GET the line string
  11. }
  12. if err := sc.Err(); err != nil {
  13. log.Fatalf("scan file error: %v", err)
  14. return
  15. }
  16. }

Read file use bufio.Reader,

  1. func readFileLines() {
  2. f, err := os.OpenFile(logfile, os.O_RDONLY, os.ModePerm)
  3. if err != nil {
  4. log.Fatalf("open file error: %v", err)
  5. return
  6. }
  7. defer f.Close()
  8. rd := bufio.NewReader(f)
  9. for {
  10. line, err := rd.ReadString('\n')
  11. if err != nil {
  12. if err == io.EOF {
  13. break
  14. }
  15. log.Fatalf("read file line error: %v", err)
  16. return
  17. }
  18. _ = line // GET the line string
  19. }
  20. }

Be aware that this bufio.Reader example will not read the last line in a file if it does not end with a newline. ReadString will return both the last line and io.EOF in this case. - konrad
The code use bufio.Reader will lost the last line of the file. if err== io.EOF it cannot break directly, that time line has the last line of the file. - Justin


  1. import (
  2. "bufio"
  3. "os"
  4. )
  5. var (
  6. reader = bufio.NewReader(os.Stdin)
  7. )
  8. func ReadFromStdin() string{
  9. result, _ := reader.ReadString('\n')
  10. witl := result[:len(result)-1]
  11. return witl
  12. }

Here is an example with function ReadFromStdin() it’s like fmt.Scan(&name) but its takes all strings with blank spaces like: “Hello My Name Is …”

  1. var name string = ReadFromStdin()
  2. println(name)

In the code bellow, I read the interests from the CLI until the user hits enter and I’m using Readline:

  1. interests := make([]string, 1)
  2. r := bufio.NewReader(os.Stdin)
  3. for true {
  4. fmt.Print("Give me an interest:")
  5. t, _, _ := r.ReadLine()
  6. interests = append(interests, string(t))
  7. if len(t) == 0 {
  8. break;
  9. }
  10. }
  11. fmt.Println(interests)

In Go 1.1 and newer the most simple way to do this is with a bufio.Scanner. Here is a simple example that reads lines from a file:

  1. package main
  2. import (
  3. "bufio"
  4. "fmt"
  5. "log"
  6. "os"
  7. )
  8. func main() {
  9. file, err := os.Open("/path/to/file.txt")
  10. if err != nil {
  11. log.Fatal(err)
  12. }
  13. defer file.Close()
  14. scanner := bufio.NewScanner(file)
  15. for scanner.Scan() {
  16. fmt.Println(scanner.Text())
  17. }
  18. if err := scanner.Err(); err != nil {
  19. log.Fatal(err)
  20. }
  21. }

This is the cleanest way to read from a Reader line by line.

There is one caveat: Scanner does not deal well with lines longer than 65536 characters. If that is an issue for you then then you should probably roll your own on top of Reader.Read().


And since the OP asked to scan over a file, it would be trivial to first file, _ := os.Open("/path/to/file.csv") and then scan over the file handle: scanner := bufio.NewScanner(file) - Evan Plumlee
Don’t forget to defer file.Close(). - Kiril
Problem is Scanner.Scan() is limited in a 4096 []byte buffer size per line. You will get bufio.ErrTooLong error, which is bufio.Scanner: token too long if the line is too long. In which case, you’ll have to use bufio.ReaderLine() or ReadString(). - eduncan911
Just my $0.02 - this is the most correct answer on the page :) - sethvargo
You can configured Scanner to handle even longer lines using its Buffer() method: golang.org/pkg/bufio/#Scanner.Buffer - Alex Robinson


EDIT: As of go1.1, the idiomatic solution is to use bufio.Scanner

I wrote up a way to easily read each line from a file. The Readln(*bufio.Reader) function returns a line (sans \n) from the underlying bufio.Reader struct.

  1. // Readln returns a single line (without the ending \n)
  2. // from the input buffered reader.
  3. // An error is returned iff there is an error with the
  4. // buffered reader.
  5. func Readln(r *bufio.Reader) (string, error) {
  6. var (isPrefix bool = true
  7. err error = nil
  8. line, ln []byte
  9. )
  10. for isPrefix && err == nil {
  11. line, isPrefix, err = r.ReadLine()
  12. ln = append(ln, line...)
  13. }
  14. return string(ln),err
  15. }

You can use Readln to read every line from a file. The following code reads every line in a file and outputs each line to stdout.

  1. f, err := os.Open(fi)
  2. if err != nil {
  3. fmt.Printf("error opening file: %v\n",err)
  4. os.Exit(1)
  5. }
  6. r := bufio.NewReader(f)
  7. s, e := Readln(r)
  8. for e == nil {
  9. fmt.Println(s)
  10. s,e = Readln(r)
  11. }

Cheers!


I wrote this answer before Go 1.1 came out. Go 1.1 has a Scanner package in the stdlib. that provides the same functionality as my answer. I would recommend using Scanner instead of my answer since Scanner is in the stdlib. Happy hacking! :-) - Malcolm


  1. // strip '\n' or read until EOF, return error if read error
  2. func readline(reader io.Reader) (line []byte, err error) {
  3. line = make([]byte, 0, 100)
  4. for {
  5. b := make([]byte, 1)
  6. n, er := reader.Read(b)
  7. if n > 0 {
  8. c := b[0]
  9. if c == '\n' { // end of line
  10. break
  11. }
  12. line = append(line, c)
  13. }
  14. if er != nil {
  15. err = er
  16. return
  17. }
  18. }
  19. return
  20. }

You can also use ReadString with \n as a separator:

  1. f, err := os.Open(filename)
  2. if err != nil {
  3. fmt.Println("error opening file ", err)
  4. os.Exit(1)
  5. }
  6. defer f.Close()
  7. r := bufio.NewReader(f)
  8. for {
  9. path, err := r.ReadString(10) // 0x0A separator = newline
  10. if err == io.EOF {
  11. // do something here
  12. break
  13. } else if err != nil {
  14. return err // if you return error
  15. }
  16. }

bufio.Reader.ReadLine() works well. But if you want to read each line by a string, try to use ReadString(‘\n’). It doesn’t need to reinvent the wheel.

ft_authoradmin  ft_create_time2020-08-27 14:22
 ft_update_time2020-08-27 14:24