Unable to cache https pages

[jcphlux]

If I turn off https on my site I am able to cache pages but the webjob fails when site is set to https.

Webjob logs
With HTTPS turned off -still gtt some errors but it will cache like 12 pages

12/27/2019 22:16:37 > 8aad20: SYS INFO] Status changed to Initializing [12/27/2019 22:16:39 > 8aad20: SYS INFO] Run script 'ProjectNamiCacheLoader.exe' with script host - 'WindowsScriptHost' [12/27/2019 22:16:40 > 8aad20: SYS INFO] Status changed to Running [12/27/2019 22:16:45 > 8aad20: INFO] Visited - http://rentprep.azurewebsites.net [12/27/2019 22:16:51 > 8aad20: INFO] Visited - http://rentprep.azurewebsites.net/ [12/27/2019 22:16:54 > 8aad20: INFO] Visited - http://rentprep.azurewebsites.net/tenant-background-check-2/ [12/27/2019 22:16:56 > 8aad20: INFO] Visited - http://rentprep.azurewebsites.net/tenant-screening/credit-check/ [12/27/2019 22:16:58 > 8aad20: INFO] Visited - http://rentprep.azurewebsites.net/tenant-screening/smartmove/ [12/27/2019 22:17:00 > 8aad20: INFO] Visited - http://rentprep.azurewebsites.net/tenant-screening/high-volume/ [12/27/2019 22:17:00 > 8aad20: INFO] ERROR Get Response - The underlying connection was closed: An unexpected error occurred on a send. at System.Net.HttpWebRequest.GetResponse() [12/27/2019 22:17:00 > 8aad20: INFO] at ProjectNamiCacheLoader.Module1.GetResponse(String LinkURL) in C:\VS2013_Projects\ProjectNamiCacheLoader\ProjectNamiCacheLoader\Module1.vb:line 112 [12/27/2019 22:17:00 > 8aad20: INFO] ERROR Unable to visit https://rentprep.azurewebsites.net/about/screening-api/ - Object reference not set to an instance of an object. at ProjectNamiCacheLoader.Module1.VisitLink(String LinkURL, Int32 LinkDepth) in C:\VS2013_Projects\ProjectNamiCacheLoader\ProjectNamiCacheLoader\Module1.vb:line 37 [12/27/2019 22:17:02 > 8aad20: INFO] Visited - http://rentprep.azurewebsites.net/tenant-screening/ [12/27/2019 22:17:05 > 8aad20: INFO] Visited - http://rentprep.azurewebsites.net/tenant-screening/sample-report/ [12/27/2019 22:17:07 > 8aad20: INFO] Visited - http://rentprep.azurewebsites.net/tenant-screening/order/ [12/27/2019 22:17:09 > 8aad20: INFO] Visited - http://rentprep.azurewebsites.net/about/ [12/27/2019 22:17:11 > 8aad20: INFO] Visited - http://rentprep.azurewebsites.net/about/contact/ [12/27/2019 22:17:13 > 8aad20: INFO] Visited - http://rentprep.azurewebsites.net/resources/ [12/27/2019 22:17:16 > 8aad20: INFO] Visited - http://rentprep.azurewebsites.net/blog/ [12/27/2019 22:17:18 > 8aad20: INFO] Visited - http://rentprep.azurewebsites.net/podcast/ [12/27/2019 22:17:20 > 8aad20: INFO] Visited - http://rentprep.azurewebsites.net/group/ [12/27/2019 22:17:22 > 8aad20: INFO] Visited - http://rentprep.azurewebsites.net/landlord-tenant-laws/ [12/27/2019 22:17:24 > 8aad20: INFO] Visited - http://rentprep.azurewebsites.net/resources/renters-relations/ [12/27/2019 22:17:26 > 8aad20: INFO] Visited - http://rentprep.azurewebsites.net/resources/faq/ [12/27/2019 22:17:28 > 8aad20: INFO] Visited - http://rentprep.azurewebsites.net/tenant-screening/packages-pricing/ [12/27/2019 22:17:28 > 8aad20: INFO] ERROR Get Response - The underlying connection was closed: An unexpected error occurred on a send. at System.Net.HttpWebRequest.GetResponse() [12/27/2019 22:17:28 > 8aad20: INFO] at ProjectNamiCacheLoader.Module1.GetResponse(String LinkURL) in C:\VS2013_Projects\ProjectNamiCacheLoader\ProjectNamiCacheLoader\Module1.vb:line 112 [12/27/2019 22:17:28 > 8aad20: INFO] ERROR Unable to visit https://rentprep.azurewebsites.net/tenant-screening/packages-pricing/ - Object reference not set to an instance of an object. at ProjectNamiCacheLoader.Module1.VisitLink(String LinkURL, Int32 LinkDepth) in C:\VS2013_Projects\ProjectNamiCacheLoader\ProjectNamiCacheLoader\Module1.vb:line 37 [12/27/2019 22:17:28 > 8aad20: INFO] ERROR Get Response - The underlying connection was closed: An unexpected error occurred on a send. at System.Net.HttpWebRequest.GetResponse() [12/27/2019 22:17:28 > 8aad20: INFO] at ProjectNamiCacheLoader.Module1.GetResponse(String LinkURL) in C:\VS2013_Projects\ProjectNamiCacheLoader\ProjectNamiCacheLoader\Module1.vb:line 112 [12/27/2019 22:17:28 > 8aad20: INFO] ERROR Unable to visit https://rentprep.azurewebsites.net/tenant-screening/ - Object reference not set to an instance of an object. at ProjectNamiCacheLoader.Module1.VisitLink(String LinkURL, Int32 LinkDepth) in C:\VS2013_Projects\ProjectNamiCacheLoader\ProjectNamiCacheLoader\Module1.vb:line 37 [12/27/2019 22:17:28 > 8aad20: INFO] ERROR Get Response - The underlying connection was closed: An unexpected error occurred on a send. at System.Net.HttpWebRequest.GetResponse() [12/27/2019 22:17:28 > 8aad20: INFO] at ProjectNamiCacheLoader.Module1.GetResponse(String LinkURL) in C:\VS2013_Projects\ProjectNamiCacheLoader\ProjectNamiCacheLoader\Module1.vb:line 112 [12/27/2019 22:17:28 > 8aad20: INFO] ERROR Unable to visit https://rentprep.azurewebsites.net/tenant-screening/rental-background-check/ - Object reference not set to an instance of an object. at ProjectNamiCacheLoader.Module1.VisitLink(String LinkURL, Int32 LinkDepth) in C:\VS2013_Projects\ProjectNamiCacheLoader\ProjectNamiCacheLoader\Module1.vb:line 37 [12/27/2019 22:17:28 > 8aad20: INFO] ERROR Get Response - The underlying connection was closed: An unexpected error occurred on a send. at System.Net.HttpWebRequest.GetResponse() [12/27/2019 22:17:28 > 8aad20: INFO] at ProjectNamiCacheLoader.Module1.GetResponse(String LinkURL) in C:\VS2013_Projects\ProjectNamiCacheLoader\ProjectNamiCacheLoader\Module1.vb:line 112 [12/27/2019 22:17:28 > 8aad20: INFO] ERROR Unable to visit https://rentprep.azurewebsites.net/tenant-screening/credit-check/ - Object reference not set to an instance of an object. at ProjectNamiCacheLoader.Module1.VisitLink(String LinkURL, Int32 LinkDepth) in C:\VS2013_Projects\ProjectNamiCacheLoader\ProjectNamiCacheLoader\Module1.vb:line 37 [12/27/2019 22:17:28 > 8aad20: INFO] ERROR Get Response - The underlying connection was closed: An unexpected error occurred on a send. at System.Net.HttpWebRequest.GetResponse() [12/27/2019 22:17:28 > 8aad20: INFO] at ProjectNamiCacheLoader.Module1.GetResponse(String LinkURL) in C:\VS2013_Projects\ProjectNamiCacheLoader\ProjectNamiCacheLoader\Module1.vb:line 112 [12/27/2019 22:17:28 > 8aad20: INFO] ERROR Unable to visit https://rentprep.azurewebsites.net/rentprep-reviews/ - Object reference not set to an instance of an object. at ProjectNamiCacheLoader.Module1.VisitLink(String LinkURL, Int32 LinkDepth) in C:\VS2013_Projects\ProjectNamiCacheLoader\ProjectNamiCacheLoader\Module1.vb:line 37 [12/27/2019 22:17:28 > 8aad20: INFO] ERROR Get Response - The underlying connection was closed: An unexpected error occurred on a send. at System.Net.HttpWebRequest.GetResponse() [12/27/2019 22:17:28 > 8aad20: INFO] at ProjectNamiCacheLoader.Module1.GetResponse(String LinkURL) in C:\VS2013_Projects\ProjectNamiCacheLoader\ProjectNamiCacheLoader\Module1.vb:line 112 [12/27/2019 22:17:28 > 8aad20: INFO] ERROR Unable to visit https://rentprep.azurewebsites.net/resources/landlord-forms/ - Object reference not set to an instance of an object. at ProjectNamiCacheLoader.Module1.VisitLink(String LinkURL, Int32 LinkDepth) in C:\VS2013_Projects\ProjectNamiCacheLoader\ProjectNamiCacheLoader\Module1.vb:line 37 [12/27/2019 22:17:28 > 8aad20: INFO] ERROR Get Response - The underlying connection was closed: An unexpected error occurred on a send. at System.Net.HttpWebRequest.GetResponse() [12/27/2019 22:17:28 > 8aad20: INFO] at ProjectNamiCacheLoader.Module1.GetResponse(String LinkURL) in C:\VS2013_Projects\ProjectNamiCacheLoader\ProjectNamiCacheLoader\Module1.vb:line 112 [12/27/2019 22:17:28 > 8aad20: INFO] ERROR Unable to visit https://rentprep.azurewebsites.net/blog/ - Object reference not set to an instance of an object. at ProjectNamiCacheLoader.Module1.VisitLink(String LinkURL, Int32 LinkDepth) in C:\VS2013_Projects\ProjectNamiCacheLoader\ProjectNamiCacheLoader\Module1.vb:line 37 [12/27/2019 22:17:28 > 8aad20: INFO] ERROR Get Response - The underlying connection was closed: An unexpected error occurred on a send. at System.Net.HttpWebRequest.GetResponse() [12/27/2019 22:17:28 > 8aad20: INFO] at ProjectNamiCacheLoader.Module1.GetResponse(String LinkURL) in C:\VS2013_Projects\ProjectNamiCacheLoader\ProjectNamiCacheLoader\Module1.vb:line 112 [12/27/2019 22:17:28 > 8aad20: INFO] ERROR Unable to visit https://rentprep.azurewebsites.net/podcast/ - Object reference not set to an instance of an object. at ProjectNamiCacheLoader.Module1.VisitLink(String LinkURL, Int32 LinkDepth) in C:\VS2013_Projects\ProjectNamiCacheLoader\ProjectNamiCacheLoader\Module1.vb:line 37 [12/27/2019 22:17:28 > 8aad20: INFO] ERROR Get Response - The underlying connection was closed: An unexpected error occurred on a send. at System.Net.HttpWebRequest.GetResponse() [12/27/2019 22:17:28 > 8aad20: INFO] at ProjectNamiCacheLoader.Module1.GetResponse(String LinkURL) in C:\VS2013_Projects\ProjectNamiCacheLoader\ProjectNamiCacheLoader\Module1.vb:line 112 [12/27/2019 22:17:28 > 8aad20: INFO] ERROR Unable to visit https://rentprep.azurewebsites.net/resources/ - Object reference not set to an instance of an object. at ProjectNamiCacheLoader.Module1.VisitLink(String LinkURL, Int32 LinkDepth) in C:\VS2013_Projects\ProjectNamiCacheLoader\ProjectNamiCacheLoader\Module1.vb:line 37 [12/27/2019 22:17:28 > 8aad20: INFO] ERROR Get Response - The underlying connection was closed: An unexpected error occurred on a send. at System.Net.HttpWebRequest.GetResponse() [12/27/2019 22:17:28 > 8aad20: INFO] at ProjectNamiCacheLoader.Module1.GetResponse(String LinkURL) in C:\VS2013_Projects\ProjectNamiCacheLoader\ProjectNamiCacheLoader\Module1.vb:line 112 [12/27/2019 22:17:28 > 8aad20: INFO] ERROR Unable to visit https://rentprep.azurewebsites.net/about/contact/ - Object reference not set to an instance of an object. at ProjectNamiCacheLoader.Module1.VisitLink(String LinkURL, Int32 LinkDepth) in C:\VS2013_Projects\ProjectNamiCacheLoader\ProjectNamiCacheLoader\Module1.vb:line 37 [12/27/2019 22:17:28 > 8aad20: INFO] ERROR Get Response - The underlying connection was closed: An unexpected error occurred on a send. at System.Net.HttpWebRequest.GetResponse() [12/27/2019 22:17:28 > 8aad20: INFO] at ProjectNamiCacheLoader.Module1.GetResponse(String LinkURL) in C:\VS2013_Projects\ProjectNamiCacheLoader\ProjectNamiCacheLoader\Module1.vb:line 112 [12/27/2019 22:17:28 > 8aad20: INFO] ERROR Unable to visit https://rentprep.azurewebsites.net/news/community/ - Object reference not set to an instance of an object. at ProjectNamiCacheLoader.Module1.VisitLink(String LinkURL, Int32 LinkDepth) in C:\VS2013_Projects\ProjectNamiCacheLoader\ProjectNamiCacheLoader\Module1.vb:line 37 [12/27/2019 22:17:28 > 8aad20: INFO] ERROR Get Response - The underlying connection was closed: An unexpected error occurred on a send. at System.Net.HttpWebRequest.GetResponse() [12/27/2019 22:17:28 > 8aad20: INFO] at ProjectNamiCacheLoader.Module1.GetResponse(String LinkURL) in C:\VS2013_Projects\ProjectNamiCacheLoader\ProjectNamiCacheLoader\Module1.vb:line 112 [12/27/2019 22:17:28 > 8aad20: INFO] ERROR Unable to visit https://rentprep.azurewebsites.net/tenant-screening/affiliate-program/ - Object reference not set to an instance of an object. at ProjectNamiCacheLoader.Module1.VisitLink(String LinkURL, Int32 LinkDepth) in C:\VS2013_Projects\ProjectNamiCacheLoader\ProjectNamiCacheLoader\Module1.vb:line 37 [12/27/2019 22:17:30 > 8aad20: INFO] Visited - http://rentprep.azurewebsites.net/tenant-screening/testimonials/ [12/27/2019 22:17:32 > 8aad20: INFO] Visited - http://rentprep.azurewebsites.net/about/security-privacy/ [12/27/2019 22:17:32 > 8aad20: INFO] ERROR Get Response - The underlying connection was closed: An unexpected error occurred on a send. at System.Net.HttpWebRequest.GetResponse() [12/27/2019 22:17:32 > 8aad20: INFO] at ProjectNamiCacheLoader.Module1.GetResponse(String LinkURL) in C:\VS2013_Projects\ProjectNamiCacheLoader\ProjectNamiCacheLoader\Module1.vb:line 112 [12/27/2019 22:17:32 > 8aad20: INFO] ERROR Unable to visit https://rentprep.azurewebsites.net/tenant-screening/questions-to-ask-potential-tenants/ - Object reference not set to an instance of an object. at ProjectNamiCacheLoader.Module1.VisitLink(String LinkURL, Int32 LinkDepth) in C:\VS2013_Projects\ProjectNamiCacheLoader\ProjectNamiCacheLoader\Module1.vb:line 37 [12/27/2019 22:17:32 > 8aad20: INFO] ERROR Get Response - The underlying connection was closed: An unexpected error occurred on a send. at System.Net.HttpWebRequest.GetResponse() [12/27/2019 22:17:32 > 8aad20: INFO] at ProjectNamiCacheLoader.Module1.GetResponse(String LinkURL) in C:\VS2013_Projects\ProjectNamiCacheLoader\ProjectNamiCacheLoader\Module1.vb:line 112 [12/27/2019 22:17:32 > 8aad20: INFO] ERROR Unable to visit https://rentprep.azurewebsites.net/tenant-screening/tenant-screening-services/ - Object reference not set to an instance of an object. at ProjectNamiCacheLoader.Module1.VisitLink(String LinkURL, Int32 LinkDepth) in C:\VS2013_Projects\ProjectNamiCacheLoader\ProjectNamiCacheLoader\Module1.vb:line 37 [12/27/2019 22:17:32 > 8aad20: INFO] ERROR Get Response - The underlying connection was closed: An unexpected error occurred on a send. at System.Net.HttpWebRequest.GetResponse() [12/27/2019 22:17:32 > 8aad20: INFO] at ProjectNamiCacheLoader.Module1.GetResponse(String LinkURL) in C:\VS2013_Projects\ProjectNamiCacheLoader\ProjectNamiCacheLoader\Module1.vb:line 112 [12/27/2019 22:17:32 > 8aad20: INFO] ERROR Unable to visit https://rentprep.azurewebsites.net/screening-services/tenant-background-check-services-reviews/ - Object reference not set to an instance of an object. at ProjectNamiCacheLoader.Module1.VisitLink(String LinkURL, Int32 LinkDepth) in C:\VS2013_Projects\ProjectNamiCacheLoader\ProjectNamiCacheLoader\Module1.vb:line 37 [12/27/2019 22:17:32 > 8aad20: SYS INFO] Status changed to Success

With HTTPS turned on - will not cache any pages

[12/27/2019 22:56:31 > 8aad20: SYS INFO] Status changed to Initializing [12/27/2019 22:56:33 > 8aad20: SYS INFO] Run script 'ProjectNamiCacheLoader.exe' with script host - 'WindowsScriptHost' [12/27/2019 22:56:33 > 8aad20: SYS INFO] Status changed to Running [12/27/2019 22:56:34 > 8aad20: INFO] ERROR Get Response - The underlying connection was closed: An unexpected error occurred on a send. at System.Net.HttpWebRequest.GetResponse() [12/27/2019 22:56:34 > 8aad20: INFO] at ProjectNamiCacheLoader.Module1.GetResponse(String LinkURL) in C:\VS2013_Projects\ProjectNamiCacheLoader\ProjectNamiCacheLoader\Module1.vb:line 112 [12/27/2019 22:56:34 > 8aad20: INFO] ERROR Unable to visit https://rentprep.azurewebsites.net - Object reference not set to an instance of an object. at ProjectNamiCacheLoader.Module1.VisitLink(String LinkURL, Int32 LinkDepth) in C:\VS2013_Projects\ProjectNamiCacheLoader\ProjectNamiCacheLoader\Module1.vb:line 37 [12/27/2019 22:56:34 > 8aad20: SYS INFO] Status changed to Success

[patrickebates]

I've never seen an issue like the one you report with HTTPS enabled. I have these jobs running many times daily on many sites, both with and without HTTPS.

As for the first problem, I think it's rather simple. You have HTTPS disabled, but it appears links are being found in your site/theme/post/etc which are hard set to HTTPS.

[jcphlux]

this is a really large site that I am migrating to azure. the All-In-One WP backup file was 2.5gig. I had to write a plugin to migrate the media file to blob storage. It could be the pages came in from the migration with hardcoded urls. I will check the tables to see if that is true.

[patrickebates]

I would add, I have stopped using the cache loader on some larger sites which are operating on Premium app service plans with dedicated vCores for SQL DB. There weren't any issues, but we determined that a significant number of pages were being placed into cache which were never being accessed. With the performance of Premium and the SQL Elastic Pool, the cost to generate the page fresh and have it stored in cache at that time was minimal.

[jcphlux]

For this site I ended up needing to do regular WP with a MySql backend. They had a very large number of plugins and I could not get ProjectNami working with it :( When I have the cache on I am definitely seeing an improvement to performance.
I will have to keep looking into the mixed links I am not seeing any permalinks in the DB everything is using the proper slugs link table for the site navigation.
I am not an expert on MySQL so maybe I will need to look into upgrading that instance.

[patrickebates]

Note that the cache does work without the loader. You just have the inital cost of generating expired pages, which then are written to the cache and the TTLs set.

How often does page content need to be regenerated? You can set the TTLs for a few hours or even longer. Plus, if a page or post is edited in WP Admin, the plugin forces a cache purge for it immediately anyway.

And as you have probably noticed, site admins can manually clear individual pages from cache by visiting them while logged in. The button will be in the Admin bar.

[jcphlux]

Yes I did notice that nice feature. I think maybe this might be an issue in the way the guid id lookup is happening on the wp_posts table. The conversion from the guid id to the permalink might not be stripping the https from the url when converting it. I noticed that it is stored in that table ref the original site subdomain it was hosted on. when I create a new page the guid is http://rentprep.azurewebsites.net/?p=<ID> and all the other guids are https://rentprep.wpengine.com/?p=<ID>

[jcphlux]

I was able to get it working by upgrading the project to .net 4.72 and update all the NuGet Packages.

[patrickebates]

I can think of one reason that would have been the solution. By any chance is only TLS 1.2 enabled on your web app? .NET 4.7 is required for TLS 1.2 to be the default HTTPS method from apps.

I will update the project here on GitHub accordingly.

[jcphlux]

Yes it is. I think it is now the default when you add a new Web App.