running the integration tests too many times causes a failure

[yosemitebandit]

The connections will timeout. @kkroo had similar issues in client and created a workaround. I wasn't able to get it working though :/

I used this script to generate the failures:
for i in {1..15}; do echo $i; nosetests openbts.tests.integration_tests:SIPAuthServeTest; done

It will fail around round 10 for me. I could restart sipauthserve with supervisorctl to get the tests to pass again.

[yosemitebandit]

or the tests will fail with InvalidRequestError: storing new value failed

[yosemitebandit]

a single test (not the whole test case) will run 99 times until it starts failing:
nosetests integration_tests:SIPAuthServeTest.test_get_all_subscribers

[yosemitebandit]

with the latest code, integration tests will run once successfully and then fail if you try to run them a second time. The problem does seem to be SIPAuthServe.

[yosemitebandit]

SIPAuthServe seems to get in a bad state and lock the sqlitedb, from /var/log/OpenBTS.log: Jun 8 18:07:28 endaga sipauthserve: ERR 14737:14745 2015-06-08T18:07:28.3 sqlite3util.cpp:62:sqlite3_prepare_statement: sqlite3_prepare_v2 failed code=14 for "update sip_buddies set port="8888" where name="IMSI000123"": unable to open database file

thankfully haven't seen this in prod yet.

[yosemitebandit]

I induced this error on a production system. I registered a sub and then ran some of the SIPAuthServe integration tests on the running box until I started getting InvalidRequestError in the tests. Then I tried a balance check with the sub and I got a message that my phone was not provisioned. I restarted SIPAuthServe with supervisor and now the balance checks work again.

This was with 0.0.18 which has the new socket handling (aggressive teardown -- here is the PR).

[yosemitebandit]

ok, this happens in production now when endaga-gprsd is running with 0.0.18 of this lib

[yosemitebandit]

I noticed this disconnect call in JSONDB.cpp. By my read it only runs when JSONDB::query is in the table.compare("subscribers") block of its big if-else tree (that is, disconnect is only after for SIPAuthServe NodeManager requests) .

In that tree (linked here), if a valid response is created it doesn't immediately get returned. This is different than the SMQueue and OpenBTS blocks, (e.g. this example) -- valid responses are immediately returned.

The consequence of not immediately returning is that disconnect is called. So that's a marked difference between the SIPAuthServe and (OpenBTS, SMQueue) components..

[yosemitebandit]

I built a version of SIPAuthServe with that disconnect line commented out. For good measure it also returns valid response immediately. I get the same integration test error as before, but where I could previously run test_subscriber_filter 41 times before a failure, I can now run that test about half as many times.

[yosemitebandit]

sudo lsof shows a huge number of open files (using this):

$ cat lsof.log | awk '{ print $2 " " $1; }' | sort -rn | uniq -c | sort -rn | head -20
8328 509 sipauthse
4023 1519 freeswitc
  73 14331 postgres
  64 24571 sshd

where the PID is 509 (sipauthserve) and I believe 8328 is the count of open files, most of which seem to be /var/lib/asterisk/sqlite3dir/sqlite3.db and sqlite3.db-wal

I'm guessing that's the reason why the sqlite db is failing to open now, though it would be nice to have JSONDB print the actual sqlite error code..

[yosemitebandit]

I think I've solved this by modifying JSONDB.cpp in NodeManager. I added more explicit disconnects, and now the open file descriptors no longer balloon, and I can run the integration tests repeatedly without socket errors.

I used openbts-python 0.0.17 for this test.