File creation of rprotoc doesn't match behavior of protoc

[codekitchen]

(copied from mozy#16)

This relates to proto files that imports other proto files. My specific use case involves having a directory hierarchy that matches the namespace hierarchy. A short solution would be for rprotoc to only output the proto file it's being asked to compile and not also generate all imports at the same time.

Initially I ran rprotoc and kept all of the files in a top level directory. Then imports fail because they are structured as 'namespace/File.pb.rb'. Copying the output into the same directory structure would give me runtime errors since both versions of the pb.rb file were being loaded.

I've found a few work arounds. Currently I'm creating the directory hierarchy and running multiple compiles, one for each file. Side effect is redundant files which I will then go and delete.

I'm currently working on Mac OS X with ruby 1.8.7, FreeBSD 8.2 with ruby 1.9 and 1.8.7. ruby_protobuf gem is 0.4.11.

Included below is the commands and output from rprotoc and protoc. Output is from FreeBSD 8.2, ruby 1.8.7, ruby_protobuf 0.4.11, protoc version 2.4.1

Related to issues #9, #10.

Directory structure.
code/gbe/pnl

$ ls -alR gbe
total 30
drwxr-xr-x  3 fuzzle  fizzle      5 Dec 28 10:58 .
drwxr-xr-x  5 fuzzle  fizzle     16 Dec 28 10:53 ..
-rw-r--r--  1 fuzzle  fizzle   3913 Dec 28 10:50 Portfolio.proto
drwxr-xr-x  2 fuzzle  fizzle      3 Dec 28 10:57 pnl

gbe/pnl:
total 6
drwxr-xr-x  2 fuzzle  fizzle     3 Dec 28 10:57 .
drwxr-xr-x  3 fuzzle  fizzle     5 Dec 28 10:58 ..
-rw-r--r--  1 fuzzle  fizzle  2149 Dec 28 10:48 Statistics.proto

# run the compiler on the top level proto file
$ rprotoc -o gbe -p ".;gbe" gbe/Portfolio.proto      
gbe/Statistics.pb.rb writing...
Warning, couldn't test load proto file because of imports
gbe/Portfolio.pb.rb writing...
$ ls -alR gbe
total 31
drwxr-xr-x  3 fuzzle  fizzle      7 Dec 28 10:59 .
drwxr-xr-x  5 fuzzle  fizzle     16 Dec 28 10:53 ..
-rw-r--r--  1 fuzzle  fizzle   6505 Dec 28 10:59 Portfolio.pb.rb
-rw-r--r--  1 fuzzle  fizzle   3913 Dec 28 10:50 Portfolio.proto
-rw-r--r--  1 fuzzle  fizzle   3864 Dec 28 10:59 Statistics.pb.rb
drwxr-xr-x  2 fuzzle  fizzle      3 Dec 28 10:57 pnl

gbe/pnl:
total 6
drwxr-xr-x  2 fuzzle  fizzle     3 Dec 28 10:57 .
drwxr-xr-x  3 fuzzle  fizzle     7 Dec 28 10:59 ..
-rw-r--r--  1 fuzzle  fizzle  2149 Dec 28 10:48 Statistics.proto
$ 

# but, now loads won't work because the import statement is:
import 'gbe/pnl/Statistics.proto';

$ ruby18 ./Portfolio.rb | less
/usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:36:in `gem_original_require': no such file to load -- gbe/pnl/Statistics.pb (LoadError)
        from /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:36:in `require'
        from ./gbe/Portfolio.pb.rb:155
        from /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:36:in `gem_original_require'
        from /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:36:in `require'
        from ./Portfolio.rb:2

# so, now go run the compiler on the sub-directory

$ rprotoc -o gbe/pnl -p ".;gbe" gbe/pnl/Statistics.proto 
gbe/pnl/Statistics.pb.rb writing...
$ ls -alR gbe
total 41
drwxr-xr-x  3 fuzzle  fizzle      7 Dec 28 10:59 .
drwxr-xr-x  5 fuzzle  fizzle     16 Dec 28 10:53 ..
-rw-r--r--  1 fuzzle  fizzle   6505 Dec 28 10:59 Portfolio.pb.rb
-rw-r--r--  1 fuzzle  fizzle   3913 Dec 28 10:50 Portfolio.proto
-rw-r--r--  1 fuzzle  fizzle   3864 Dec 28 10:59 Statistics.pb.rb
drwxr-xr-x  2 fuzzle  fizzle      4 Dec 28 11:02 pnl

gbe/pnl:
total 7
drwxr-xr-x  2 fuzzle  fizzle     4 Dec 28 11:02 .
drwxr-xr-x  3 fuzzle  fizzle     7 Dec 28 10:59 ..
-rw-r--r--  1 fuzzle  fizzle  3864 Dec 28 11:02 Statistics.pb.rb
-rw-r--r--  1 fuzzle  fizzle  2149 Dec 28 10:48 Statistics.proto


# now the program will run, if done as this example
$ ruby18 ./Portfolio.rb 
...correct output...
$

# but, now there are multiple Statistics outputs located in the build system. Some imports will fail because they find both Statistics.pb.rb files
# and the numbers collide.


##### Using the protoc compiler ######

# the initial directory layout
$ ls -alR gbe
total 30
drwxr-xr-x  3 fuzzle  fizzle      5 Dec 28 11:05 .
drwxr-xr-x  5 fuzzle  fizzle     16 Dec 28 10:53 ..
-rw-r--r--  1 fuzzle  fizzle  20480 Dec 28 10:51 .Portfolio.proto.swp
-rw-r--r--  1 fuzzle  fizzle   3913 Dec 28 10:50 Portfolio.proto
drwxr-xr-x  2 fuzzle  fizzle      3 Dec 28 11:05 pnl

gbe/pnl:
total 6
drwxr-xr-x  2 fuzzle  fizzle     3 Dec 28 11:05 .
drwxr-xr-x  3 fuzzle  fizzle     5 Dec 28 11:05 ..
-rw-r--r--  1 fuzzle  fizzle  2149 Dec 28 10:48 Statistics.proto
[fuzzle@gbe-jamdev-02 ~/work/pjm/sample_code/ruby_protobuf]$ 

# running protoc
$ protoc --cpp_out=. --proto_path . --proto_path gbe  gbe/Portfolio.proto 
$ ls -alR gbe
total 31
drwxr-xr-x  3 fuzzle  fizzle       7 Dec 28 11:11 .
drwxr-xr-x  5 fuzzle  fizzle      16 Dec 28 10:53 ..
-rw-r--r--  1 fuzzle  fizzle  116777 Dec 28 11:11 Portfolio.pb.cc
-rw-r--r--  1 fuzzle  fizzle   76031 Dec 28 11:11 Portfolio.pb.h
-rw-r--r--  1 fuzzle  fizzle    3913 Dec 28 10:50 Portfolio.proto
drwxr-xr-x  2 fuzzle  fizzle       3 Dec 28 11:05 pnl

gbe/pnl:
total 6
drwxr-xr-x  2 fuzzle  fizzle     3 Dec 28 11:05 .
drwxr-xr-x  3 fuzzle  fizzle     7 Dec 28 11:11 ..
-rw-r--r--  1 fuzzle  fizzle  2149 Dec 28 10:48 Statistics.proto

# NOTE: Above in the directory structure we were putting the output files into . and they ended up in gbe. The protoc
# compiler implicitly assumes that you'll have a directory structure similar to the namespace.
# Also, there is only the output for the file that we asked it to compile, not the file + imports.

# running the command on the statistics file.
$ protoc --cpp_out=. --proto_path . --proto_path gbe  gbe/pnl/Statistics.proto 
$ ls -alR gbe
total 220
drwxr-xr-x  3 fuzzle  fizzle       7 Dec 28 11:11 .
drwxr-xr-x  5 fuzzle  fizzle      16 Dec 28 10:53 ..
-rw-r--r--  1 fuzzle  fizzle  116777 Dec 28 11:11 Portfolio.pb.cc
-rw-r--r--  1 fuzzle  fizzle   76031 Dec 28 11:11 Portfolio.pb.h
-rw-r--r--  1 fuzzle  fizzle    3913 Dec 28 10:50 Portfolio.proto
drwxr-xr-x  2 fuzzle  fizzle       5 Dec 28 11:13 pnl

gbe/pnl:
total 7
drwxr-xr-x  2 fuzzle  fizzle      5 Dec 28 11:13 .
drwxr-xr-x  3 fuzzle  fizzle      7 Dec 28 11:11 ..
-rw-r--r--  1 fuzzle  fizzle  55621 Dec 28 11:13 Statistics.pb.cc
-rw-r--r--  1 fuzzle  fizzle  37536 Dec 28 11:13 Statistics.pb.h
-rw-r--r--  1 fuzzle  fizzle   2149 Dec 28 10:48 Statistics.proto

# now the files are created and can be imported. 
# NOTE: we asked for the output to be in '.' and the Statistics files ended up in gbe/pnl which matches the namespace and our import statements.

# Similar output is obtained if you run using python_out.