Christmas Time its been shorter visits

I have been cut back on visits to see Duke. Im hoping its not because Im worried about him. Hes been working with Krystyn and myself on the off day of the week (Wed Tues or Thurs) when Krystyn has some extra time.  Two weeks ago I was explaining to Krystyn of how Duke does not like to be ridden and especially in the arena he veers to the left.  I tacked him before she got there. The month of Nov to Dec was terribly cold and snow eadh week sometimes 2 storms 3 snows in one week. This makes it more diffcult to get to the barn.  Well I tacked him we worked on mounting in the arena an then I got on, Duke had done good. Well I went around one time and he looked very uncomfortable. He scaredmme with his actions.  I sometimes thinks he may not likeme  but it comces down to the fact that maybe hes in pain. Maybe his saddle is very uncomfortable on him and he back is not up to being ridden.
Well This week I as was working with him, he was particularly pushy, kicking nearme.  when I was working with Krystyn she reminded me that I need to not let that happen. This stuck in my head, why was he doing this? Was I not hard enough on him? I think I need to use the whip.
He is ok with the treetment and I like how Krystyn incorporates it into the training.
I feel like Im missing out on the ground work that we used to do and then I would walk him on the road.
Yesterday the stupid car was racing its motor made me uncomfortable.
Should I be walking him on the road but I have since stopped doing that.
For Christmas I order him a treeless saddle that is treeless and its an endurance saddle.  Krystyn had said that June used one on her horse and that it was helpful. It was on Africa the older horse.
Idlike to stop using my current status on his working.
He cxrowds me in the arena hes fine to handle around the farm. I think Ill put in my cleats and walk around outside. Avoiding the arena may be better for me.
Last week I was getting him ready for his picture and he was so jumpy iwth th Alpacas. I blamed them but I still concerned that its not really the alpacas but hes just too jumpy.