GSoC Proposal: Improvements to the REPL workflow.

[soumitradev]

Hi, my name is Soumitra Shewale, and I'm a first year dual degree Economics student (second degree choice pending), at Birla Institute of Technology and Science Pilani, Hyderabad. I have worked extensively with Julia before (in Google Code-in 2019) and I learnt a lot during that period. When I found out about GSoC, I immediately checked if Julia was in the list, and sure enough, I found a project I was interested in.

I have prior experience with Julia and Images.jl, and I have some experience with cross compiling on linux, and I am also sufficiently familiar with C. I will admit however, that I have no experience in wrapping C libraries in Julia. Maybe I can learn a thing or two during the course of this project?

I do have a few questions regarding this project however:

• Is native Julia code preferred over a wrapped library?
• Are other supported image formats using a wrapped library? If so, why?
• Which version(s) of EXIF do we plan to support? Can we augment the existing libexif library to support versions newer than 2.2, or will we have to stick to native code for that?

Looking forward to a great discussion (and possibly project 😉). Thanks!

[johnnychen94]

Hi Soumitra, this project is also planned by another student @ashwani-rathee #1000. Usually, we don't have multiple students selected for the same project idea so I would recommend you to come up with some other ideas.

Other similar projects I have in mind are:

1. wrapping the ImageMagick v7 project. The current ImageMagick.jl builds upon ImageMagick v7 and it's known problematic. While we're trying to provide ImageIO a successor, ImageMagick does support a lot of formats that ImageIO may never support.
2. improving the OpenCV.jl project based on the previous GSoC results in https://github.com/archit120/OpenCV.jl. Most of the infrastructure on the C++ side is finished, but there is still a lot of room to improve the Julia-side experiences.

Let us know if you're interested in this, or if you have other ideas related to JuliaImages.

[soumitradev]

Hmm... I noticed #1000 mentions two tasks: one being the EXIF one, and the other being the JPEG2000 and GIF one. Maybe one possible way forward is for me to take the JPEG2000 and GIF one since they've already started work on the EXIF one?

If not, I'd be interested in the first task you mentioned. Would that require me to transfer this discussion to ImageMagick.jl?

However, I did notice you've been interested in replacing ImageMagick with ImageIO. Is there any other functionality that ImageIO lacks but is provided by ImageMagick? I think that would move us a bit more forward in regards to your goal for ImageIO.

Hope you can guide me through this. Thanks!

[johnnychen94]

I noticed #1000 mentions two tasks: one being the EXIF one, and the other being the JPEG2000 and GIF one.

Yes, I've discussed this with him and we believe EXIF might only make a 120 hour project, so we add JPEG2000 and GIF to make it a 350 hour project.

However, I did notice you've been interested in replacing ImageMagick with ImageIO. Is there any other functionality that ImageIO lacks but is provided by ImageMagick?

This is mainly for performance consideration; we constantly find out wrapping the actual C backend libraries provides 1.5-3x performance improvement over ImageMagick. Thus for commonly used image formats, we would want to make it efficient enough. From the documentation in https://imagemagick.org/script/formats.php, there are more than 100 formats not supported by ImageIO. With this great coverage, we might never get a chance to really have ImageIO support that many formats, and thus we might never have a chance to actually sunset it. It is impossible and not necessary to have ImageIO support these many formats.

Also, you might have seen a lot of legacy issues in https://github.com/JuliaIO/ImageMagick.jl. Instead of fixing it on the legacy ImageMagick v6, I would prefer a rewrite on top of ImageMagick v7. That should automatically fix a lot of issues.

we constantly find out wrapping the actual C backend libraries provides 1.5-3x performance improvement over wrapping ImageMagick.

See also the benchmarks in JuliaIO/ImageIO.jl#21 and JuliaIO/JpegTurbo.jl#15

[johnnychen94]

Another choice is to cover other missing formats similar to #1000, such as APNG, BMP, (HDR)RGBE, and other PNG variants.

[johnnychen94]

There's also another idea that I would like to see people tackle: build a terminal-based IDE to further improve the current REPL workflow via Term.jl + UnicodePlots + ImageInTerminal. I haven't done concept proof yet but with Term.jl I believe this is doable.

I believe @t-bltg (who currently maintains UnicodePlots) would also be interested in this project. I would also want to invite @FedeClaudi to share some thoughts on whether this is doable.

[FedeClaudi]

Hi,

That sounds cool!

build a terminal-based IDE to further improve the current REPL workflow

I haven't used Images.jl much so I'm not sure what this refers to, though I'm more than happy to advice on how Term can be used for whatever Terminal needs you have.

[johnnychen94]

I assume we can use Term to build a layout manager, and then let UnicodePlots and ImageInTerminal prints to the Term-based IO output. For instance, I'm imagine something like this

The current ImageInterminal looks like this:

[FedeClaudi]

That's so cool! It would be awesome to integrate that with Term!

[soumitradev]

This is mainly for performance consideration; we constantly find out wrapping the actual C backend libraries provides 1.5-3x performance improvement over ImageMagick.

Oh yeah, I think I understand a bit better now.

Also, you might have seen a lot of legacy issues in https://github.com/JuliaIO/ImageMagick.jl. Instead of fixing it on the legacy ImageMagick v6, I would prefer a rewrite on top of ImageMagick v7. That should automatically fix a lot of issues.

The only issue I have with wrapping ImageMagick is that it might be a project I wouldn't be able to complete. I imagine wrapping ImageMagick v7 is around a 350 hour project (especially since it has been framed as a 'rewrite'), and I want to pick a small to medium sized (~175-200hrs) project. I'm afraid I'll bite off more than I can chew.

There's also another idea that I would like to see people tackle: build a terminal-based IDE to further improve the current REPL workflow via Term.jl + UnicodePlots + ImageInTerminal. I haven't done concept proof yet but with Term.jl I believe this is doable.

This sounds like a really fun idea! I've never used the Images.jl REPL workflow, but I imagine I can familiarize myself with it in a couple of days and look at some potential concrete plans of improving it. Would I be considered qualified for the job though?

[johnnychen94]

I'm a bit busy finishing my PhD, so I'm weary of taking on additional commitments :)

Same here. Even though I still get excited when I see new people contribute to Julia and want to spend my limited spare time on it :)

Mentoring a GSoC project means that you want to supervise the project to help it go in a good direction by providing feedback (as you're doing right now) and code reviews. There will be a mentor linked to the GSoC admin to fill the paperwork, I can be that man if you find it too busy.

[FedeClaudi]

Same here. Even though I still get excited when I see new people contribute to Julia and want to spend my limited spare time on it :)

Yeah!

I'm happy to pitch in to help with Term, but I no nothing about Images.jl so I think I'm not the best person to mentor this project.

[soumitradev]

When updating the console, you'll need to clear it and then print out all the content. The process should look something like:

1. clear the console

2. create the renderables/layout you'll need to display

3. print them with `print` or `tprint`

Yes that's exactly what I'm planning to do.

What I'd recoment is to use an IOBuffer to write all the stuff from steps 1,2&3 and only when done with everything empty the IOBuffer content onto stdout or where ever you're printing stuff. I do something like this for progressbars.

So I'm in the process of trying this currently, and when I try to run the REPL on my custom IOBuffers, I get:

MethodError: no method matching check_open(::IOBuffer)

Full log of what I tried:

julia> using REPL

julia> inbuf = IOBuffer(); outbuf = IOBuffer(); errbuf = IOBuffer();

julia> term = REPL.Terminals.TTYTerminal("dumb", inbuf, outbuf, errbuf)
REPL.Terminals.TTYTerminal("dumb", IOBuffer(data=UInt8[...], readable=true, writable=true, seekable=true, append=false, size=0, maxsize=Inf, ptr=1, mark=-1), IOBuffer(data=UInt8[...], readable=true, writable=true, seekable=true, append=false, size=0, maxsize=Inf, ptr=1, mark=-1), IOBuffer(data=UInt8[...], readable=true, writable=true, seekable=true, append=false, size=0, maxsize=Inf, ptr=1, mark=-1))

julia> repl = REPL.LineEditREPL(term, true)
LineEditREPL(REPL.Terminals.TTYTerminal("dumb", IOBuffer(data=UInt8[...], readable=true, writable=true, seekable=true, append=false, size=0, maxsize=Inf, ptr=1, mark=-1), IOBuffer(data=UInt8[...], readable=true, writable=true, seekable=true, append=false, size=0, maxsize=Inf, ptr=1, mark=-1), IOBuffer(data=UInt8[...], readable=true, writable=true, seekable=true, append=false, size=0, maxsize=Inf, ptr=1, mark=-1)), true, "\e[32m", "\e[0m", "\e[0m", "\e[31m", "\e[33m", false, false, false, false, false, nothing, REPL.Options(true, Dict{Any, Any}[], 8, 100, 0.2, 0.2, 0.2, 1.0, ["\e[90m"], true, true, true, false, true, false, false, 0.005, 0.05, Dict{Symbol, Any}()), nothing, Tuple{String, Int64}[], #undef, #undef)

julia> REPL.run_repl(repl)
Unhandled Task ERROR: MethodError: no method matching check_open(::IOBuffer)
Closest candidates are:
  check_open(::Union{Base.LibuvServer, Base.LibuvStream}) at C:\Users\Soumi\AppData\Local\Programs\Julia-1.7.1\share\julia\base\stream.jl:384
  check_open(::Base.Filesystem.File) at C:\Users\Soumi\AppData\Local\Programs\Julia-1.7.1\share\julia\base\filesystem.jl:115
Stacktrace:
 [1] raw!(t::REPL.Terminals.TTYTerminal, raw::Bool)
   @ REPL.Terminals C:\Users\Soumi\AppData\Local\Programs\Julia-1.7.1\share\julia\stdlib\v1.7\REPL\src\Terminals.jl:127
 [2] prompt!(term::REPL.Terminals.TextTerminal, prompt::REPL.LineEdit.ModalInterface, s::REPL.LineEdit.MIState)
   @ REPL.LineEdit C:\Users\Soumi\AppData\Local\Programs\Julia-1.7.1\share\julia\stdlib\v1.7\REPL\src\LineEdit.jl:2571
 [3] run_interface(terminal::REPL.Terminals.TextTerminal, m::REPL.LineEdit.ModalInterface, s::REPL.LineEdit.MIState)
   @ REPL.LineEdit C:\Users\Soumi\AppData\Local\Programs\Julia-1.7.1\share\julia\stdlib\v1.7\REPL\src\LineEdit.jl:2488
 [4] run_frontend(repl::LineEditREPL, backend::REPL.REPLBackendRef)
   @ REPL C:\Users\Soumi\AppData\Local\Programs\Julia-1.7.1\share\julia\stdlib\v1.7\REPL\src\REPL.jl:1230
 [5] (::REPL.var"#49#54"{LineEditREPL, REPL.REPLBackendRef})()
   @ REPL .\task.jl:423
ERROR: TaskFailedException
Stacktrace:
  [1] try_yieldto(undo::typeof(Base.ensure_rescheduled))
    @ Base .\task.jl:777
  [2] wait()
    @ Base .\task.jl:837
  [3] wait(c::Base.GenericCondition{ReentrantLock})
    @ Base .\condition.jl:123
  [4] take_buffered(c::Channel{Any})
    @ Base .\channels.jl:389
  [5] take!
    @ .\channels.jl:383 [inlined]
  [6] repl_backend_loop(backend::REPL.REPLBackend)
    @ REPL C:\Users\Soumi\AppData\Local\Programs\Julia-1.7.1\share\julia\stdlib\v1.7\REPL\src\REPL.jl:239
  [7] start_repl_backend(backend::REPL.REPLBackend, consumer::Any)
    @ REPL C:\Users\Soumi\AppData\Local\Programs\Julia-1.7.1\share\julia\stdlib\v1.7\REPL\src\REPL.jl:229
  [8] run_repl(repl::AbstractREPL, consumer::Any; backend_on_current_task::Bool)
    @ REPL C:\Users\Soumi\AppData\Local\Programs\Julia-1.7.1\share\julia\stdlib\v1.7\REPL\src\REPL.jl:362
  [9] run_repl (repeats 2 times)
    @ C:\Users\Soumi\AppData\Local\Programs\Julia-1.7.1\share\julia\stdlib\v1.7\REPL\src\REPL.jl:349 [inlined]
 [10] top-level scope
    @ REPL[5]:1

    nested task error: MethodError: no method matching check_open(::IOBuffer)
    Closest candidates are:
      check_open(::Union{Base.LibuvServer, Base.LibuvStream}) at C:\Users\Soumi\AppData\Local\Programs\Julia-1.7.1\share\julia\base\stream.jl:384
      check_open(::Base.Filesystem.File) at C:\Users\Soumi\AppData\Local\Programs\Julia-1.7.1\share\julia\base\filesystem.jl:115
    Stacktrace:
     [1] raw!(t::REPL.Terminals.TTYTerminal, raw::Bool)
       @ REPL.Terminals C:\Users\Soumi\AppData\Local\Programs\Julia-1.7.1\share\julia\stdlib\v1.7\REPL\src\Terminals.jl:127
     [2] prompt!(term::REPL.Terminals.TextTerminal, prompt::REPL.LineEdit.ModalInterface, s::REPL.LineEdit.MIState)
       @ REPL.LineEdit C:\Users\Soumi\AppData\Local\Programs\Julia-1.7.1\share\julia\stdlib\v1.7\REPL\src\LineEdit.jl:2571
     [3] run_interface(terminal::REPL.Terminals.TextTerminal, m::REPL.LineEdit.ModalInterface, s::REPL.LineEdit.MIState)
       @ REPL.LineEdit C:\Users\Soumi\AppData\Local\Programs\Julia-1.7.1\share\julia\stdlib\v1.7\REPL\src\LineEdit.jl:2488
     [4] run_frontend(repl::LineEditREPL, backend::REPL.REPLBackendRef)
       @ REPL C:\Users\Soumi\AppData\Local\Programs\Julia-1.7.1\share\julia\stdlib\v1.7\REPL\src\REPL.jl:1230
     [5] (::REPL.var"#49#54"{LineEditREPL, REPL.REPLBackendRef})()
       @ REPL .\task.jl:423

julia>

I read through some documentation, and it looks like the function to run the REPL is looking for a Base.TTY, but the IOBuffer doesn't implement the needed functionality. I'm not exactly sure about the type hierarchy with regards to them, so I'll do a bit more digging and see what I can find. In the mean time, if any of you know what I'm doing wrong, feel free to point out :)

Redirect the in_stream and out_stream should be the best solution I can think of.

Yup, I agree too.

You can start with the "refreshing at a fixed rate" idea to quickly put things together, and then try to improve the performance by only updating when needed. Does this sound better to you?

Sure! For the proof of concept, I'll refresh 30 times a second.

It totally depends on how long you want to spend on this GSoC project; both are okay to me.

Some of my plans for the summer changed, and I'd love to work on a 350-hour-sized project, but I'm not sure if the current scope of this project would allow for it, and I might have to come up with more features to implement. That said, I don't know how long each of the tasks we laid out will take, and I think in the next few days when I'm drafting the proposal, I'll have more clarity on this. For now however, since we marked the debugger as optional, the assumption that this is a medium (175-hour-sized) project would be a safe one.

[FedeClaudi]

Maybe this helps?

import Term: inspect

inspect(Base.IOBuffer)
inspect(Base.TTY)

[soumitradev]

Great! Looks like they're type-cousins. Looks like they don't have documentation in the help prompt, so I'll read through their implementations, and what makes one work in a bit.

Edit: Some of the testing code here looks pretty useful 👀

[soumitradev]

So I read through the test code I linked, and I used Pipes instead of IOBuffers since they inherit from LibuvStream (which is what the REPL wants, and I think I'm getting closer. I can create a REPL that doesn't display anything, and I can control the input and output pipes pretty well, but it still has some issues.

Here's a simplified version of the test code I was using (this one has all the REPL hooks removed, so what we have is just a REPL running in the background, we have no control over how expressions are evaluated):

using REPL

inputbuf = Pipe()
outputbuf = Pipe()
errbuf = Pipe()

Base.link_pipe!(inputbuf, reader_supports_async=true, writer_supports_async=true)
Base.link_pipe!(outputbuf, reader_supports_async=true, writer_supports_async=true)
Base.link_pipe!(errbuf, reader_supports_async=true, writer_supports_async=true)

term = REPL.Terminals.TTYTerminal("dumb", inputbuf.out, outputbuf.in, errbuf.in)
repl = REPL.LineEditREPL(term, true)

@async begin
  REPL.run_repl(repl)
end

commands = [
  "\n",
  "a = 1\n",
  "println(a)\n",
  "b = a/2\n",
  "b\n",
  "exit()\n",
]

for cmd in commands
  println("STDIN:")
  writeDone = @async begin
    write(inputbuf.in, cmd)
  end

  wait(writeDone)
  println("STDOUT:")
  outReadDone = @async begin
    println(String(readavailable(outputbuf.out)))
  end

  wait(outReadDone)
  # println("STDERR:")
  # errReadDone = @async begin
  #   println(String(readavailable(errbuf.out)))
  # end

  # wait(errReadDone)
  println("--------------")
  sleep(5)
end

Output:

STDIN:
STDOUT:
julia>

--------------
STDIN:
STDOUT:

julia> a =
--------------
STDIN:
STDOUT:
julia> a = 1
1

julia> pri
--------------
1
STDIN:
STDOUT:
julia> println(a)

julia> b =
--------------
STDIN:
STDOUT:
julia> b = a/2
0.5

julia> b

--------------
STDIN:
STDOUT:
0.5

julia> exi
--------------

Some issues with this however:

• When there is no error, and our error pipe is empty, the read on it blocks, and the next command can't be executed.
• The custom stdout pipe is garbled, and seems to have both the input and the output of the REPL.

What am I doing wrong here? I presume it's something with how I'm reading/writing from/to the Pipes, or some of the async stuff. Hope you can help. Thanks!

[johnnychen94]

I'll check the code later this weekend.