Category Archives: Technical

06.27.16
by

Connections Upgrade UI Weirdness

My latest PMR for a Connections 5.5 upgrade was concerning some very strange behaviour in the UI.  Seemingly unrelated but looking suspiciously like the same cause. Here’s what we discovered (and yes it was tested with multiple IDs , browsers and locations)

  1. The public files window had no scroll bar
  2. There were no upload buttons on the Files app
  3. When editing a profile picture I couldn’t remove it, only change it.  If I chose remove it just closed the profile with no changes.  The wsadmin profile command to remove the picture worked fine
  4. Editing a Community I owned just showed a blank page with a Community title but not editable fields

Weird right?

No errors anywhere in SystemOut or even in filebug or fiddler.  IBM could find no problems in the configuration or errors anywhere so today we did a screenshare to see what was going on.  After 90 minutes we had no luck, not even an error so Charlie Price joined the call as we talked over what I did to build the environment I explained it was a side by side upgrade where I

  • Built a new 5.5 server
  • Upgraded it to CR1
  • migrated the data to 5.0 CR1 databases using dbt.jar
  • upgraded those databases first to 5.5 then to CR1
  • used the migration tool to export artifacts from 5.0
  • copied the work folder the migration tool created to the new 5.5 server
  • used migration lc-import (from the updated tool issued D1 Dec 18) to import the artifacts

When I looked at the customization folder after completing the migration it was full of content that had been imported so the first thing I did when I saw this problem was empty that folder (by copying it somewhere else) and getting it to match my Connections101 5.5 CR1 server.  My suspicion was that somehow the migration lc-import had broken something.

Charlie compared with his system and noticed that both the javascript folder under Customizations and the webresources folder under provision were both completely wrong.  When we checked the 5.0 environment we saw that the contents matched that so must have been overwritten during migration lc-import.  The clue was the missing javascript and jar files for Ephox which I had installed and were now missing.  Charlie sent me a zipped up copy of his webresources directory and I overwrote mine with that and everything started working.

So.. be very careful using the migration wizard.  Take copies of your javascript and provision\webresources directories before doing an import and then make sure the files look up to date and there is nothing missing when you’re done.

 

 

 

06.20.16
by

Connections Failure To Authenticate

Last week was spent working on a PMR where a newly migrated (side by side) Connections 5.5 environment refused to let anyone access any applications.  I could login using any credential but the Homepage wouldn’t load and any application that required authentication failed including Communities.

Here are some of the errors in the logs

CLFRW0016E: Could not retrieve details for the user with login ID gabriella.davis@domainname.com due to an exception. The exception occurred when retrieving the details via the virtual member manager directly: {1} (in system out for utilcluster which contains homepage)

ADMN0022E: Access is denied for the expandVariable operation on AdminOperations MBean because of insufficient or empty credentials. (in ffdc)

“CustomAuthent E com.ibm.connections.httpClient.CustomAuthenticatorFactory <init> SONATA: authenticator class name is missing!  {in SystemOut for InfraCluster)

webapp E com.ibm.ws. webcontainer.webapp.WebApp logServletError SRVE0293E: [Servlet Error]- [action]: com.ibm.tango.exception.AuthContextException: com.ibm. connections.directory.services.exception.DSException: com.ibm. connections.directory.services.exception.DSOutOfServiceException: java. lang.NullPointerException (in Systemout for InfraCluster).

 

Here (amongst others) are the things we tested / changed / reverted that didn’t fix it.  Bear in mind a working 5.0 production environment with the exact same configuration had no problems during this time.

  • LDAP was fine (we could login). For giggles we changed credentials and back again
  • We changed the login options from mail;cn;uid (which we use in this environment and works fine) to uid;mail;cn
  • We removed the mapped credentials for application security that were put there by the installer and put them back again – apparently that sometimes works
  • Set the authentication under application security for Communities and Profiles from None to Everyone just to confirm where the problem was
  • About 100 other things

Basically we managed to establish the issue was any intraservice communication but not why.  Eventually it went to L3 who isolated the error  as being something in the LotusConnections-Config.xml.

CustomAuthent E com.ibm.connections.httpClient.CustomAuthenticatorFactory <init> SONATA: authenticator class name is missing!  

That file had been migrated as an artifact via the migration tool and was the same as 5.0 but in there was the line <tns:customAuthenticator name=”DefaultAuthenticator” xmlns:tns1=”http://www.ibm.com/uiextensions-config”/>;

which they asked to be changed to <customAuthenticator name=”DefaultAuthenticator”/>

That immediately fixed the problem.

No-one is quite sure how that setting ever got into LotusConnections-Config.xml but my guess is during a CCM/Filenet installation.  The interesting thing is that it works in 5.0 but breaks 5.5. Maybe it requires you to have CCM installed to work as the 5.5 environment (mine or IBMs) didn’t have that.

Still a nice simple fix for such a painful problem and maybe somewhere for you to check when doing your own debugging.

Thanks very much to David McCarthy & the IBM L2 team for prioritising and working the problem.

06.14.16
by

Severe TDI Issue In Connections 5.5

I have been working with a customer who is migrating to Connections 5.5 from Connections 5.0.   When I do a migration I like to do it properly and create clean data by using dbt.jar to migrate content to new databases.  I know a lot of people are happy with the backup/restore of databases idea but for me that leaves too much scope for bad data to carry over from old system to new.

Everything was going fine, the profiles data migrated and then I tried a sync all dns to sync the ldap data to the database.  Something we schedule daily at this customer.  It failed as it tried to hash the database tables.  The error in the ibmdi.log was

Error: The sort page size property – source_ldap_sort_page_size= – must be greater than 10 if it is not 0. Aborting.

That’s a value that is set in profiles_tdi.properties and it was already set to 0.  So why was it aborting?

I decided to troubleshoot just with a cutdown list of names in collect.dns and using populate_from_dn_file function.  Again it failed but with the strangest error that would find the user in LDAP, get all their values then fail to find the user in the database and fail to update.

In SyncUpdates.log I could see the following error no matter what user I chose for populate_from_dn_file.

ERROR [com.ibm.di.log.FileRollerAppender.bc9c35a0-aae5-416e-9a99-1d418c3c564c] – [callSyncDB_mod] [ProfileConnector] null
java.lang.IndexOutOfBoundsException
at java.util.Collections$EmptyList.get(Collections.java:87)

I then tried copying the collect.dns to my 5.0 production environment and running there and it worked fine, found the users as duplicates and didn’t update them which is the correct behaviour.

I compared the map_dbrepos_from_source.properties files in 5.0 with 5.5 and it all looked pretty much the same.  So I opened a PMR which was eventually escalated to development. As soon as they received it they knew what the problem was – apparently a known but not documented bug that was fixed in CR1 with files that you have to manually deploy (we were already at CR1).

Development’s report of the problem was

log4j.logger.com.ibm.lconn.profiles.internal.service=ALL           
                                                                       
in log4j.properties causes TDI populate and sync commands to crash if an
EMPLOYEE is altered        

Well the crashing was true but the value  log4j.logger.com.ibm.lconn.profiles.internal.service=ALL was # out and unused so it wasn’t related to that particular log setting in my case.

The fix was to go find the two files

lc.profiles.core.service.impl.jar
lc.profiles.core.service.api.jar

in the Connections install and copy them to your TDI\lib directory in your tdisol environment.  In my case I had created a folder called TDISOL55 and under that I had a TDI directory with all the properties, script etc files in and the lib subdirectory full of jar files.  That came from the D1 (day 1) release download of Wizards which contained updated TDIPopulation directory and was dated 18th Dec.  There was no new tdisol with CR1 but clearly there should have been.

I found the files in my Websphere Application profile directory for the profiles application server under the directory

D:\IBM\WebSphere\AppServer\profiles\AppSrv01\installedApps\conn55Cell01\Profiles.ear

I copied those two files over and it al-most worked.  I had one more problem.  The value source_ldap_sort_attribute in profiles_tdi.properties which was initially set to empty (not null but = ) had been changed at the request of L2 to source_ldap_sort_attribute=mail which matched the 5.0 properties we were using.  They asked me to change it for exact comparison and that broke the updates.  Once I took out the “mail” mapping the scripts, both populate_from_dn_file and sync_all_dns ran perfectly.

The new environment does use different LDAP servers (but the same source data) and I don’t know if attempting to tell the server to sort the LDAP results failing is a problem with that server configuration (both environments are Domino 9.0.1) or 5.5 itself. I’ll investigate that and update.

So my two fixes were

  • copy the two jar files from your CR1 installedapps directory to your TDISol directory (lib subdirectory).
  • make sure source_ldap_sort_attribute= in profiles_tdi.properties
06.06.16
by

Sametime For Mobile Users – #NWTL

My final New Way To Learn session today was looking at the Sametime mobile clients, Connections Chat and Sametime Meetings.  I hope you find it useful and as always the full recorded session is available in the #NWTL Community.

The slides by themselves are below

In this session we looked at the architecture behind the Sametime mobile applications for chat and meetings. What do you need to deploy to support mobile users and what features are available to them on the different mobile platforms. We also looked at potential bottlenecks, security and troubleshooting for the mobile clients.

06.03.16
by

Sametime Audio and Video For External Users – NWTL

Today I did the second in my series of Sametime presentations for IBM’s New Way To Learn (NWTL) initiative.    The session was recorded with audio and is available by joining the Community here http://bit.ly/1t7e0LE . The session slides by themselves are on slideshare and shown below.

If your Sametime environment is going to include Audio and Video you will probably want to be able to talk to people outside your own company, or at least to your own users on their mobile devices who aren’t connected via VPN. In this recorded online session as part of IBM’s New Way To Work initiative we reviewed the infrastructure behind the Audio and Video elements of Sametime and how best to extend those features beyond your firewall.

 

03.30.16
by

Sametime WAS Proxy Stops Working

I’ve had an interesting system down call with an existing Sametime 9.0.1 customer in the past week.  The environment is over 18 months old and consists of every server component in single instances including ST Proxy, Meetings, ST Advanced and all Media components.  The media components were added in Dec 2015 and everything has been fine. The Meeting and Proxy servers both have WAS proxies in front of them to handle traffic over port 80 / 443 separately.  Last week the Meeting node was restarted and the WAS Proxy stopped working.  It would load.  The Meeting server was responding on its own application ports to http(s)://hostname:9080 / 9443 both worked but http(s)://hostname failed with

503 Service Unavailable

The WAS Proxy server showed started.  There were no errors in the logs for that or the ST Meeting server.  Not all WAS proxies were broken because the one in front of the ST Proxy server worked.  In short that error suggests that the Meeting server is offline when we knew it wasn’t and since there isn’t any real configuration for the WAS Proxy other than what node it points to – there was nothing to troubleshoot.  I tried deleting and recreating the WAS Proxy a few times, I tried switching it to use alternate ports 81/444, nothing would fix it.

It took a few days and some combined effort to find.  The WAS team wanted us to upgrade to WAS fixpack 5 but that would mean upgrading 8 working servers in the hopes of fixes one WAS proxy.  There was a suggestion that since the Meeting server was a single, not a cluster, I could just change the Meeting server ports to use 80/443 instead of 9080/9443 and do away with the WAS proxy entirely.  That would get rid of the problem but not fix it, just circumvent it.  I wanted to fix it and find out why it happened.

I had checked the virtual hosts to make sure the hostname / port combination was in the stmeet host and wasn’t anywhere else and discovered that in default_host new wildcard port entries had appeared for ports 80 and 443.  I had already deleted those but that didn’t fix the problem.  How did those port entries appear ? I’ve seen this before when you install new ST servers (as we did with Media in Dec) it come sometimes write virtual host entries to the wrong places.  In fact that was my first guess but after I removed those entries from default_host and it still didn’t fix the problem I was out of ideas.  Then Tony Payne from IBM spotted that the admin_host virtual host which is only used by the SSC had the ports 9080 and 9443 in it when it should only have 8700 and 8701.  Again I assume these were added by the previous server installs and of course I never went to look there because the Meeting server was specifically set to use the STMeet host.

I removed those extra ports from the admin_host virtual host definition and restarted the Meeting node and servers (clearing the temp directories first \profilename\temp and \profilename\wstemp as well as \profilename\config\temp) and that fixed the problem.

So why was the presence of those two ports 9080/9443  (used by the ST Meeting server) that were in a virtual host the ST Meeting server doesn’t even use causing the WAS Proxy to break? Why didn’t the Meeting server itself break and why didn’t the ST Proxy Server which also had a WAS proxy in front of it break?

Turns out that no matter what virtual host mapping you have in place for applications, in Sametime the code checks the admin_host and if a port appears there – it silently disables looking up any other host.  The fact that the Meeting server ports appeared at all in the admin_host meant that the STMeet host was being ignored and the WAS Proxy had no way to direct the traffic.

Unfortunately none of that is visible in the logs or in debug logs which all reported the servers and services using the correct STMeet host.  So it wasn’t something that was able to be seen.  It was a combination of Tony seeing the admin entries and me having had a previous call with a server install which added ports to unwanted virtual hosts that allowed us to find it and fix it.

The ST Proxy server itself wasn’t affected because that server was running on 9082/9445 so its ports weren’t in admin_host and its virtual host therefore wasn’t ignored.

Always good to have a problem fixed and learn a ton of stuff about application behaviour at the same time 🙂

03.30.16
by

Last week in Eindhoven…

We were in Eindhoven last week at the Engage conference.. over 400 attendees, speakers and IBM’ers gathered for two days of learning, talking and cleaning out the hotel bar of tonic water.. I’ve been to several of the past Engage conferences and Theo always puts on a great event but this was bigger and better than ever.  So why?

IBM sent a lot of executives to Engage with the Opening General Session being given by the new ICS general manager (appointed at Connect in January) Inhi Cho Suh and with product strategy presented by Suzanne Livingston , Sara Gibbons and Chris Crummey.  The first thing Inhi announced was that things are going to change – starting with the Orlando conference which moves to February 22nd at Moscone West in San Francisco.  That’s a big decision and commitment – serious tech companies have conferences in SF and that’s where ICS (IBM Collaboration Services) need to be if they are going to innovate, lead and grow as opposed to maintain.   Inhi also let us know that she has asked the product team to work on a 2020 strategy and that it will include IBM Verse on premise.

Then we got the demo of Verse , Toscana and the thinking behind ICS design.  It’s a shame the OGS wasn’t recorded as Suzanne’s background to their design thinking and Sara & Chris’ demo were both much more detailed (and further advanced) than at Connect in January.  However if you want some idea of what we saw take a look at the OGS video from January (from about 90 seconds in to 20 mins in) here

Aside from the OGS the entire IBM team (of which there were more than 30 in attendance) were everywhere wanting to hear about problems, wanting to listen, wanting to change their relationship with partners, with customers with development for the better.   It’s hard not to be taken up with the positivity and enthusiasm.  I’m an optimistic person but I don’t consider myself naive – I feel that I recognise honesty and intent when people talk to me and I what I heard that ICS was important, investable and part of the core IBM development strategy.

In short I choose to believe until I’m proved wrong.

There were of course plenty of great sessions to attend and, as usual, I missed many of the ones I wanted.  Partly because there were also lots of round table discussions too which I found very interesting.  Apparently I’m still the 8 year old in class first to put her hand up with a question.

My session on SHA2 and SSL vulnerabilities was against Mat Newman’s User Blast and Sara Gibbons’ with Toscana.   We were all along the same corridor and I watched person after person go past my room on their way to Mat or Sara’s , so thank you to everyone who chose to hear about security instead and filled out my room.  I hope you found it useful  (and the hand puppets helpful).  For anyone who wasn’t there I have added it to slideshare 

On the final evening of the event Theo invited speakers to a dinner preceded by a surprise.  The surprise was that 32 of us were sent into the Escape Rooms.. you are locked in a themed room for an hour and have to decode lots of puzzles to find the code to get out.  I’ve always wanted to try an Escape Room and I chose the “Tomb” which was an Egyptian tomb and went in with a team including Tim and Mike, Sue Smith, Bill Malchisky, Mat Newman, Rene Winkelmeyer and Carl Tyler.  We didn’t make it out in time – we were soooooo close.. but a few things to bear in mind

  • The tomb was entirely dark except for a small flashlight Tim found hidden in a basket in a corner and some candles.  My night vision varies from “bad” to “crappy”
  • Having multiple alpha males in a small space all shouting instructions at each other may not be the best way to get out quickly
  • There was sand everywhere.  Everywhere.  My shoes may never recover
  • Tim is great at puzzles but apparently in the dark, without his glasses (which he forgot to bring in) and with 7 people shouting at him to hurry up – not so much
  • There was a really cool effect where we completed a puzzle and lasers appeared out of the eyes of a skull on the wall and we had to position 7 different mirrors around the room to bounce the lasers around to hit a small hole on the wall.  We got so excited doing that we didn’t notice we had completed the puzzle and a new “door” had opened for about 10 mins.
  • I was given a cryptex to decode and open.  I broke it by pulling the end off.
  • With only 1 light source we could only do one thing at a time so some of us spent a lot of time kneeling in the sand feeling around fake skeletons for clues

In the end it was great fun and I’d definitely want to do it again.

All of that plus a chance to talk to lots of customers and see lots of friends – some of which came along just to meet up.

I hope you’re recovered Theo – because we’re all up to do it again next year.

 

 

 

 

 

10.01.15
by

Determining Connections Versions

As I start a new Healthcheck today I thought I’d share a tip with you.  One of the first things I do when coming clean to someone’s Connections environment is try and determine what’s installed, including CRs and fixes.  Installation Manager is good at telling you what it installed but less so if you installed fixes outside of its interface.  There are other methods too like checking the version logs and reading the about.jsp, but it can be fiddly to piece together all the information.

One of the best resources I’ve found is an IBM technote from this July which shows how to identify exactly what fixes and CRs are installed.  The most comprehensive is updateSilent which produces a report on screen of every version, CR and iFix.  Here’s the table of what each utility can do.

Identifying Connections

The updateSilent utility is run from with the updaterInstaller directory under your Connections install and the command is:

sh updateSilent (bat or sh depending on your OS) -fix -installDir <ConnectionsInstallDir>

You may have to set WAS_HOME first before it will run so my commands in Linux are:

WAS_HOME=/opt/IBM/WebSphere/AppServer

export WAS_HOME

sh updateSilent.sh -fix -installDir /opt/IBM/Connections

It will then output to screen every CR and iFix that exists.

09.20.15
by

Living iOS9 For A Few Days

I upgraded both my phone and iPad to iOS9 a few days’ ago but I was in a hurry and I really haven’t had a chance to review the features so I’ve just been living with the new OS and seeing what works for me and what doesn’t.  I use my iPad for Citrix, SSH & RDP, reading, watching TV and playing games.  My iPhone I use for managing my day, emails, reminders, tasks and photos.

I’d be interested to know anyone else’s favourite features that you’re actually using

The Good

  1. Popup blocker in a browser.  Yes I know blah blah editorial sites need to make money blah blah.  It makes my browser faster and sites more interesting.  I do pay for some sites to access but if the sites didn’t overdo the ads then I’d tolerate them.  But they don’t, they overdo and make the entire site unreadable.  I’ve tried Blockr and Peace (now withdrawn) and i’m definitely noticing improvements
  2. Swipe in.  On my iPad I can swipe in from the right to look at my mail whatever else i’m doing
  3. Picture in Picture – i absolutely love the PIP feature that Hulu, iPlayer, Netflix all have.  Now when I’m exercising and watching a program or cooking and watching a program I can come out of the app and do other things whilst the app shows in a small window.  LOVE

    PIP

  4. Photo syncing.  Finally my photos auto sync immediately across devices – previously it was hit and miss and could take a while for photos from my phone to reach any other device. That may have just been me because Tim never had that problem
  5. Spotlight search – customised searching across apps including streaming services.
  6. Mobile data failover when wifi is weak.  The idea is that if you are connected to wifi but it’s not strong it will switch to using mobile data instead.  I do have one dead spot in my house and usually I have to disconnect wifi so it uses data there but this fixed that problem.  It only works if you have unlimited data though.
  7. Siri seems improved, at least it seems to understand me better and write emails / texts more accurately
  8. Public transport in Apple Maps.  I’ve already used this a few times – asking for a route home last night from Hampton Court it showed me the nearest bus stops, stations , bus numbers to take and when they were due.  It only works right now in London and a few other cities but luckily that’s where i live.

The Bad

  1. Swipe In.  When reading a book every time I go to turn the page it swipes in and shows me my inbox. No Apple BAD. I don’t want to be checking my inbox when I”m trying to read.  There’s no way to turn off right in swipe so I need to reprogram my fingers to touch for page turn slightly further in from the right or use the left touch to turn the page. It’s annoying but I wouldn’t give up swipe, I just wish I could turn it off when in iBooks
  2. Spotlight search – by default all apps are included in spotlight search including “Spotlight suggestions” and “Bing Search” both of which send your query to a remote site.  I disabled those.  Also by default when I go to search it shows what book i’m reading and how far i’ve progressed.  Not a fan.  I had to go through and turn off all the apps I didn’t want searched. I would have preferred to turn “on” rather than turn “off” so another one where the feature is good but the implementation doesn’t work for me
  3. Tim absolutely HATES the new keyboard which I don’t notice at all so that’s his problem 🙂
09.15.15
by

A Statement From IBM On El Capitan and iOS9 Support

IBM have today released a statement explaining why some applications will be unable to connect to Domino servers from iOS9 and El Capitan devices due to Apple removing support for Elliptic curve technology (no – me either) and enhanced transport security.  This doesn’t affect only IBM but it’s something you need to be aware of.  There will be an interim fix for Domino 9.0.1 FP4 and also a new FP5 to resolve these issues (eta end Sept) but there will be no fix for Domino 8.5.x servers.

The full statement and explanation is here but the key summary is

Additionally, IBM is working on an Interim Fix for 9.0.1 Fix Pack 4 (and the upcoming 9.0.1 Fix Pack 5) that will implement Elliptic Curve cipher support for TLS 1.2 and TLS 1.0 that remedies this issue and implements Elliptic Curve support for the following protocols: HTTP/HTTPS, LDAP/LDAPS, SMTP, IMAP, and POP3. Currently, the ETA for the Interim Fix posting is end of September 2015.

Elliptic Curve support will not be available for Domino 8.5.x releases since the specification requires updated cryptographic libraries that are available only in Domino 9.0 and above.